[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:04 dist-test-slave-15hz krb5kdc[8254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:04 dist-test-slave-15hz krb5kdc[8254](info): set up 2 sockets
Sep 25 14:58:04 dist-test-slave-15hz krb5kdc[8254](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260925 14:58:05.385324 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.944s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:05.498705 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:45177
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:05.577014 8267 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:05.577237 8267 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:05.577260 8267 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:05.578382 8267 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:05.578421 8267 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:05.578434 8267 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:05.578445 8267 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:05.578455 8267 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:05.579964 8267 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:45177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8267
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:05.580186 8267 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:05.580390 8267 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:05.585421 8267 server_base.cc:1034] running on GCE node
W20260925 14:58:05.585578 8275 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:05.585487 8273 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:05.585495 8272 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:05.585956 8267 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:05.586223 8267 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:05.587383 8267 hybrid_clock.cc:648] HybridClock initialized: now 1790348285587368 us; error 26 us; skew 500 ppm
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:05.589918 8267 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:05.590444 8267 webserver.cc:460] Webserver started at http://127.8.13.254:36477/ using document root <none> and password file <none>
I20260925 14:58:05.590641 8267 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:05.590685 8267 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:05.590785 8267 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:05.591928 8267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ae731da729ef48d38fa21a3162ea18dd"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "b65f43861f516cf5111a1a9e196894a6"
server_key_iv: "8097248753fbd6ca2da473159b801444"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.592260 8267 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ae731da729ef48d38fa21a3162ea18dd"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "b65f43861f516cf5111a1a9e196894a6"
server_key_iv: "8097248753fbd6ca2da473159b801444"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.593626 8267 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:05.594344 8282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.594524 8267 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:05.594615 8267 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ae731da729ef48d38fa21a3162ea18dd"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "b65f43861f516cf5111a1a9e196894a6"
server_key_iv: "8097248753fbd6ca2da473159b801444"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.594678 8267 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:05.602371 8267 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:05.604413 8267 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:05.604523 8267 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:05.608356 8267 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:45177
I20260925 14:58:05.608398 8334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:45177 every 8 connection(s)
I20260925 14:58:05.608727 8267 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:05.609563 8335 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:05.611825 8335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: Bootstrap starting.
I20260925 14:58:05.612389 8335 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:05.612643 8335 log.cc:824] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:05.613293 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8267
I20260925 14:58:05.613400 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:05.613401 8335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: No bootstrap required, opened a new log
I20260925 14:58:05.614192 8247 external_mini_cluster.cc:1428] Setting key 9c7569ac357b46df3b3030b43342be8c
I20260925 14:58:05.615419 8335 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } }
I20260925 14:58:05.615621 8335 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:05.615658 8335 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae731da729ef48d38fa21a3162ea18dd, State: Initialized, Role: FOLLOWER
I20260925 14:58:05.615768 8335 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } }
I20260925 14:58:05.615828 8335 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:05.615846 8335 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:05.615891 8335 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:05.616719 8335 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } }
I20260925 14:58:05.616806 8335 leader_election.cc:302] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae731da729ef48d38fa21a3162ea18dd; no voters:
I20260925 14:58:05.616948 8335 leader_election.cc:288] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:05.616987 8340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:05.617159 8340 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [term 1 LEADER]: Becoming Leader. State: Replica: ae731da729ef48d38fa21a3162ea18dd, State: Running, Role: LEADER
I20260925 14:58:05.617214 8335 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:05.617291 8340 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } }
I20260925 14:58:05.617673 8340 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae731da729ef48d38fa21a3162ea18dd. Latest consensus state: current_term: 1 leader_uuid: "ae731da729ef48d38fa21a3162ea18dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } } }
I20260925 14:58:05.617748 8340 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:05.618860 8355 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:05.618913 8355 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:05.618957 8349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:05.619112 8349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:05.617707 8341 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae731da729ef48d38fa21a3162ea18dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae731da729ef48d38fa21a3162ea18dd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 45177 } } }
I20260925 14:58:05.619652 8341 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:05.620460 8357 mvcc.cc:205] Tried to move back new op lower bound from 7333266577899016192 to 7333266577888841728. Current Snapshot: MvccSnapshot[applied={T|T < 7333266577899016192}]
I20260925 14:58:05.620847 8349 catalog_manager.cc:1347] Generated new cluster ID: 757854c2ee9642b5ac2feef4e54b230a
I20260925 14:58:05.620906 8349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:05.632187 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:05.615267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.8.13.254:45177)
0925 14:58:05.615455 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:05.615466 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:05.615672 (+ 206us) server_negotiation.cc:407] Connection header received
0925 14:58:05.617668 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:05.617673 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:05.617733 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:05.617862 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:05.619632 (+ 1770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.621043 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.622517 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.623123 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.623213 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:05.627676 (+ 4463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:05.627680 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:05.627681 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:05.627694 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:05.629663 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:05.630223 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:05.630226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:05.630226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:05.630338 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:05.630802 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:05.630803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:05.630803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:05.631106 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:05.631247 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:05.631627 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 14:58:05.631766 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 14:58:05.646243 8349 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:05.647014 8349 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:05.669284 8349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae731da729ef48d38fa21a3162ea18dd: Generated new TSK 0
I20260925 14:58:05.669489 8349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:05.702463 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:05.784397 8361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:05.784600 8361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:05.784624 8361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:05.785825 8361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:05.785871 8361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:05.785898 8361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:05.787410 8361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:05.787662 8361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:05.787876 8361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:05.792622 8369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:05.792671 8367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:05.792671 8366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:05.792843 8361 server_base.cc:1034] running on GCE node
I20260925 14:58:05.793037 8361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:05.793290 8361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:05.794432 8361 hybrid_clock.cc:648] HybridClock initialized: now 1790348285794410 us; error 37 us; skew 500 ppm
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:05.796964 8361 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:05.797582 8361 webserver.cc:460] Webserver started at http://127.8.13.193:43337/ using document root <none> and password file <none>
I20260925 14:58:05.797813 8361 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:05.797858 8361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:05.797986 8361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:05.799081 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "4954efec361b4bd6a7378a59de45424a"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "d0175f906b3d018a1abdd85ac742945f"
server_key_iv: "5d6da39b88d70fe7614a5c232c212c67"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.799418 8361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "4954efec361b4bd6a7378a59de45424a"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "d0175f906b3d018a1abdd85ac742945f"
server_key_iv: "5d6da39b88d70fe7614a5c232c212c67"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.800726 8361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:05.801367 8376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.801545 8361 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.801622 8361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "4954efec361b4bd6a7378a59de45424a"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "d0175f906b3d018a1abdd85ac742945f"
server_key_iv: "5d6da39b88d70fe7614a5c232c212c67"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.801693 8361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:05.821913 8361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:05.824159 8361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:05.824304 8361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:05.824537 8361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:05.824860 8361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:05.824894 8361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.824923 8361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:05.824940 8361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.831136 8361 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:46567
I20260925 14:58:05.831189 8489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:46567 every 8 connection(s)
I20260925 14:58:05.831576 8361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:05.837574 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8361
I20260925 14:58:05.837678 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:05.837822 8247 external_mini_cluster.cc:1428] Setting key fa3d75ba41172ba03097f270ed68be75
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:05.843492 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:05.832934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:35039 (local address 127.8.13.254:45177)
0925 14:58:05.833001 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:05.833003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:05.833014 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:05.833689 (+ 675us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:05.833691 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:05.833708 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:05.833774 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:05.835132 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.835870 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.838017 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.838232 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.838292 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:05.840619 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:05.840621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:05.840621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:05.840628 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:05.841956 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:05.842369 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:05.842370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:05.842370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:05.842417 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:05.842739 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:05.842741 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:05.842741 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:05.842869 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:05.842932 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:05.843308 (+ 376us) server_negotiation.cc:299] Negotiation successful
0925 14:58:05.843404 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:05.843752 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:05.832637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:45177 (local address 127.8.13.193:35039)
0925 14:58:05.832830 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:05.832884 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:05.833517 (+ 633us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:05.833858 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:05.833863 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:05.834631 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:05.834956 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:05.834965 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.835979 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.835980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:05.837323 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:05.837327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.838321 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.838322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:05.838497 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:05.839084 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:05.839093 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:05.840513 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:05.842050 (+ 1537us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:05.842058 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:05.842061 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:05.842307 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:05.842470 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:05.842471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:05.842472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:05.842617 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:05.842947 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:05.842955 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:05.843146 (+ 191us) client_negotiation.cc:769] Sending connection context
0925 14:58:05.843275 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:58:05.843466 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260925 14:58:05.844358 8490 heartbeater.cc:347] Connected to a master server at 127.8.13.254:45177
I20260925 14:58:05.844453 8490 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:05.844626 8490 heartbeater.cc:511] Master 127.8.13.254:45177 requested a full tablet report, sending...
I20260925 14:58:05.845232 8299 ts_manager.cc:194] Registered new tserver with Master: 4954efec361b4bd6a7378a59de45424a (127.8.13.193:46567)
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:05.846160 8299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:35039
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:05.862600 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:05.944974 8495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:05.945182 8495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:05.945204 8495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:05.946318 8495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:05.946352 8495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:05.946376 8495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:05.947891 8495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:05.948089 8495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:05.948303 8495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:05.954840 8501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:05.955119 8500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:05.955132 8495 server_base.cc:1034] running on GCE node
W20260925 14:58:05.955313 8503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:05.955574 8495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:05.955801 8495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:05.956947 8495 hybrid_clock.cc:648] HybridClock initialized: now 1790348285956932 us; error 28 us; skew 500 ppm
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:05.959663 8495 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:05.960191 8495 webserver.cc:460] Webserver started at http://127.8.13.194:40421/ using document root <none> and password file <none>
I20260925 14:58:05.960415 8495 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:05.960460 8495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:05.960593 8495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:05.961670 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "495195d483cf4cbf9e775dbf8613026c"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "2e18980ce4e255b82a5bc1bf612b7984"
server_key_iv: "69280b8304ac0c8689c41fe69330d4ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.962023 8495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "495195d483cf4cbf9e775dbf8613026c"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "2e18980ce4e255b82a5bc1bf612b7984"
server_key_iv: "69280b8304ac0c8689c41fe69330d4ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.963285 8495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:05.964061 8510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.964251 8495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.964344 8495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "495195d483cf4cbf9e775dbf8613026c"
format_stamp: "Formatted at 2026-09-25 14:58:05 on dist-test-slave-15hz"
server_key: "2e18980ce4e255b82a5bc1bf612b7984"
server_key_iv: "69280b8304ac0c8689c41fe69330d4ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:05.964417 8495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:05.986951 8495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:05.989135 8495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:05.989261 8495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:05.989492 8495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:05.989794 8495 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:05.989826 8495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.989847 8495 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:05.989871 8495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:05.995664 8495 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:41891
I20260925 14:58:05.995723 8623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:41891 every 8 connection(s)
I20260925 14:58:05.996037 8495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:05.997644 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8495
I20260925 14:58:05.997766 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:05.997953 8247 external_mini_cluster.cc:1428] Setting key 0432b226cec87f920071eb954b0153ae
Sep 25 14:58:05 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348285, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:06.006378 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:05.996707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:41979 (local address 127.8.13.254:45177)
0925 14:58:05.996755 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:05.996757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:05.997003 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:58:05.997484 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:05.997486 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:05.997502 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:05.997553 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:05.998761 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.999257 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.000505 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.000805 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.000883 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.003152 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.003153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.003154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.003163 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.004949 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.005374 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.005375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.005375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.005436 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.005692 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.005693 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.005694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.005836 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.005904 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.006192 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.006283 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:06.006553 8626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:05.996693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:45177 (local address 127.8.13.194:41979)
0925 14:58:05.996861 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:05.996901 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:05.997399 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:05.997635 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:05.997640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:05.998408 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:05.998673 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:05.998678 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:05.999356 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:05.999357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.000410 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:06.000412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.000886 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.000886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.000981 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.001476 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:06.001484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:06.003046 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:06.005057 (+ 2011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.005063 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.005066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.005295 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.005502 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.005502 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.005503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.005645 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.005895 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:06.005898 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:06.006063 (+ 165us) client_negotiation.cc:769] Sending connection context
0925 14:58:06.006163 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 14:58:06.006296 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260925 14:58:06.007051 8624 heartbeater.cc:347] Connected to a master server at 127.8.13.254:45177
I20260925 14:58:06.007148 8624 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:06.007320 8624 heartbeater.cc:511] Master 127.8.13.254:45177 requested a full tablet report, sending...
I20260925 14:58:06.007753 8299 ts_manager.cc:194] Registered new tserver with Master: 495195d483cf4cbf9e775dbf8613026c (127.8.13.194:41891)
I20260925 14:58:06.008108 8299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:41979
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:06.023173 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:06.104393 8629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:06.104719 8629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:06.104799 8629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:06.106065 8629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:06.106112 8629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:06.106150 8629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:06.107811 8629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:45177
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:06.108059 8629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:06.108289 8629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:06.113219 8637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.113214 8635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:06.113325 8634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:06.113621 8629 server_base.cc:1034] running on GCE node
I20260925 14:58:06.113824 8629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:06.114060 8629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:06.115206 8629 hybrid_clock.cc:648] HybridClock initialized: now 1790348286115191 us; error 27 us; skew 500 ppm
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:06.117785 8629 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:06.118389 8629 webserver.cc:460] Webserver started at http://127.8.13.195:33605/ using document root <none> and password file <none>
I20260925 14:58:06.118621 8629 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:06.118669 8629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:06.118798 8629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:06.119910 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "59d0674fbfc645c3b103034bfeecb284"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-15hz"
server_key: "631eda4e33abbbd7672750d756a78c3f"
server_key_iv: "b8d3fc9b0c528dde5423c2df81981123"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.120239 8629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "59d0674fbfc645c3b103034bfeecb284"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-15hz"
server_key: "631eda4e33abbbd7672750d756a78c3f"
server_key_iv: "b8d3fc9b0c528dde5423c2df81981123"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.121475 8629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:06.122216 8644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.122390 8629 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:06.122467 8629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "59d0674fbfc645c3b103034bfeecb284"
format_stamp: "Formatted at 2026-09-25 14:58:06 on dist-test-slave-15hz"
server_key: "631eda4e33abbbd7672750d756a78c3f"
server_key_iv: "b8d3fc9b0c528dde5423c2df81981123"
server_key_version: "encryptionkey@0"
I20260925 14:58:06.122532 8629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:06.130204 8629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:06.132222 8629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:06.132351 8629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:06.132540 8629 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:06.132829 8629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:06.132861 8629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.132889 8629 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:06.132915 8629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:06.138357 8629 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:43343
I20260925 14:58:06.138405 8757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:43343 every 8 connection(s)
I20260925 14:58:06.138695 8629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:06.147400 8339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.139403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:47343 (local address 127.8.13.254:45177)
0925 14:58:06.139454 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:06.139456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:06.139660 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:58:06.140173 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:06.140174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:06.140185 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:06.140246 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:06.141181 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.141680 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.142623 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.142813 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.142875 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.144928 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.144929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.144930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.144939 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.146187 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.146541 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.146543 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.146543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.146594 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.146810 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.146811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.146812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.146922 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.146975 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.147236 (+ 261us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.147311 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:06.147609 8760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.139363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:45177 (local address 127.8.13.195:47343)
0925 14:58:06.139540 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:06.139581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:06.140071 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:06.140311 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:06.140315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:06.140848 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:06.141034 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:06.141039 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.141748 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.141749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.142537 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:06.142538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.142892 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.142893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:06.142995 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.143485 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:06.143493 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:06.144838 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:06.146263 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.146269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.146271 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.146474 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.146654 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:06.146654 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:06.146655 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:06.146754 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:06.146980 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:06.146987 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:06.147117 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:58:06.147204 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:58:06.147355 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260925 14:58:06.148079 8758 heartbeater.cc:347] Connected to a master server at 127.8.13.254:45177
I20260925 14:58:06.148180 8758 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:06.148209 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8629
I20260925 14:58:06.148275 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:06.148368 8758 heartbeater.cc:511] Master 127.8.13.254:45177 requested a full tablet report, sending...
I20260925 14:58:06.148432 8247 external_mini_cluster.cc:1428] Setting key 4934f064198191fd4d0d7afd7c8da615
I20260925 14:58:06.148741 8299 ts_manager.cc:194] Registered new tserver with Master: 59d0674fbfc645c3b103034bfeecb284 (127.8.13.195:43343)
I20260925 14:58:06.149171 8299 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:47343
I20260925 14:58:06.149435 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:06.170900 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.163452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.8.13.193:46567)
0925 14:58:06.163651 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:06.163656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:06.163664 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:06.163687 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:06.163689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:06.163702 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:06.163763 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:06.164316 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.165071 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.165799 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.166113 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.166171 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.168351 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.168359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.168359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.168366 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.169797 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.170135 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.170138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.170138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.170203 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.170386 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.170387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.170387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.170536 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.170661 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.170725 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.170790 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":108,"threads_started":1}
W20260925 14:58:06.171288 8404 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48962
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348286, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:06.192488 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:06.185526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48966 (local address 127.8.13.193:46567)
0925 14:58:06.185591 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:06.185592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:06.185603 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:06.185682 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:06.185683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:06.185696 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:06.185740 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:06.186280 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.186762 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.187487 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:06.187762 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:06.187821 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:06.190111 (+ 2290us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:06.190112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:06.190112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:06.190121 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:06.191418 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.191790 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.191791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.191792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.191842 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:06.192008 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:06.192009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:06.192009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:06.192118 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:06.192223 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:06.192313 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 14:58:06.192396 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:06.193035 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8361
I20260925 14:58:06.197079 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8495
I20260925 14:58:06.200901 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8629
I20260925 14:58:06.204551 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8267
2026-09-25T14:58:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1789 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8772](info): set up 2 sockets
Sep 25 14:58:06 dist-test-slave-15hz krb5kdc[8772](info): commencing operation
krb5kdc: starting...
W20260925 14:58:07.146854 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:07.258505 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42161
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42161
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.337144 8785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.337353 8785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.337383 8785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.338497 8785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:07.338537 8785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.338555 8785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.338570 8785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:07.338585 8785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:07.340049 8785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:07.340294 8785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.340498 8785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.345029 8790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.345125 8791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.345125 8793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.345264 8785 server_base.cc:1034] running on GCE node
I20260925 14:58:07.345496 8785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.345764 8785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.346912 8785 hybrid_clock.cc:648] HybridClock initialized: now 1790348287346894 us; error 28 us; skew 500 ppm
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.349261 8785 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:07.349742 8785 webserver.cc:460] Webserver started at http://127.8.13.254:44983/ using document root <none> and password file <none>
I20260925 14:58:07.349920 8785 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.349957 8785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.350049 8785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.351015 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "0f77e5a9a0374edf80ccee55bd754969"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "7c0bcb61feafdca1ca8710260cea2cd9"
server_key_iv: "6cd1ac136dd18386339f6318a83b7325"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.351311 8785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "0f77e5a9a0374edf80ccee55bd754969"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "7c0bcb61feafdca1ca8710260cea2cd9"
server_key_iv: "6cd1ac136dd18386339f6318a83b7325"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.352608 8785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:07.353286 8800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.353462 8785 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:07.353549 8785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "0f77e5a9a0374edf80ccee55bd754969"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "7c0bcb61feafdca1ca8710260cea2cd9"
server_key_iv: "6cd1ac136dd18386339f6318a83b7325"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.353616 8785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.364909 8785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.366844 8785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.366998 8785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.371011 8785 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42161
I20260925 14:58:07.371039 8852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42161 every 8 connection(s)
I20260925 14:58:07.371387 8785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:07.372247 8853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:07.372991 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8785
I20260925 14:58:07.373076 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:07.373221 8247 external_mini_cluster.cc:1428] Setting key 5621e14bd485f68be0ad3a0c26c006f3
I20260925 14:58:07.374719 8853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969: Bootstrap starting.
I20260925 14:58:07.375253 8853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:07.375504 8853 log.cc:824] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:07.376224 8853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969: No bootstrap required, opened a new log
I20260925 14:58:07.377876 8853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } }
I20260925 14:58:07.378027 8853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:07.378060 8853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f77e5a9a0374edf80ccee55bd754969, State: Initialized, Role: FOLLOWER
I20260925 14:58:07.378194 8853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } }
I20260925 14:58:07.378276 8853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:07.378307 8853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:07.378357 8853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:07.378851 8853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } }
I20260925 14:58:07.378932 8853 leader_election.cc:302] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f77e5a9a0374edf80ccee55bd754969; no voters:
I20260925 14:58:07.379091 8853 leader_election.cc:288] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:07.379122 8858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:07.379222 8858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [term 1 LEADER]: Becoming Leader. State: Replica: 0f77e5a9a0374edf80ccee55bd754969, State: Running, Role: LEADER
I20260925 14:58:07.379299 8858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } }
I20260925 14:58:07.379336 8853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:07.379616 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f77e5a9a0374edf80ccee55bd754969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } } }
I20260925 14:58:07.379684 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:07.379812 8858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f77e5a9a0374edf80ccee55bd754969. Latest consensus state: current_term: 1 leader_uuid: "0f77e5a9a0374edf80ccee55bd754969" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f77e5a9a0374edf80ccee55bd754969" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42161 } } }
I20260925 14:58:07.379859 8858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:07.380049 8863 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:07.380759 8863 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:07.382354 8863 catalog_manager.cc:1347] Generated new cluster ID: e3b0a6ccb5bc4cee8651c12ddc4c0182
I20260925 14:58:07.382403 8863 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:07.384048 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.373818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59776 (local address 127.8.13.254:42161)
0925 14:58:07.373998 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.374008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.374162 (+ 154us) server_negotiation.cc:407] Connection header received
0925 14:58:07.374606 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.374613 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.374660 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.374802 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.375390 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.376566 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.377276 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.377673 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.377765 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.380357 (+ 2592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.380362 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.380362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.380375 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.382478 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.382798 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.382801 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.382801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.382899 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.383080 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.383081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.383081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.383334 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.383497 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.383660 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.383881 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260925 14:58:07.425215 8863 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:07.425968 8863 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:07.431664 8863 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f77e5a9a0374edf80ccee55bd754969: Generated new TSK 0
I20260925 14:58:07.431878 8863 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:07.470628 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.564177 8879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.564394 8879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.564419 8879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.565611 8879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.565651 8879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.565677 8879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:07.567167 8879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:07.567365 8879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.567565 8879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.572273 8884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.572398 8879 server_base.cc:1034] running on GCE node
W20260925 14:58:07.572261 8887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.572243 8885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.572705 8879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.572935 8879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.574077 8879 hybrid_clock.cc:648] HybridClock initialized: now 1790348287574048 us; error 42 us; skew 500 ppm
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.576560 8879 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:07.577059 8879 webserver.cc:460] Webserver started at http://127.8.13.193:41405/ using document root <none> and password file <none>
I20260925 14:58:07.577268 8879 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.577314 8879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.577437 8879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.578449 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "95420da8fd994e318c7f6bb14a6a5289"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "a850976443d92117f895129102f9d6c6"
server_key_iv: "88eabf0327b3a4d658e973c60b8308d6"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.578778 8879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "95420da8fd994e318c7f6bb14a6a5289"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "a850976443d92117f895129102f9d6c6"
server_key_iv: "88eabf0327b3a4d658e973c60b8308d6"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.580183 8879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:07.580868 8894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.581004 8879 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:07.581079 8879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "95420da8fd994e318c7f6bb14a6a5289"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "a850976443d92117f895129102f9d6c6"
server_key_iv: "88eabf0327b3a4d658e973c60b8308d6"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.581146 8879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.609047 8879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.611042 8879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.611155 8879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.611334 8879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:07.611600 8879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:07.611640 8879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.611662 8879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:07.611677 8879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.617080 8879 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:41729
I20260925 14:58:07.617136 9007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:41729 every 8 connection(s)
I20260925 14:58:07.617409 8879 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:07.625928 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 8879
I20260925 14:58:07.626022 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:07.626228 8247 external_mini_cluster.cc:1428] Setting key 827abd4e69f30b3dd2bf38bb28d3fcec
I20260925 14:58:07.626765 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.618164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:50453 (local address 127.8.13.254:42161)
0925 14:58:07.618229 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.618231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.618408 (+ 177us) server_negotiation.cc:407] Connection header received
0925 14:58:07.619357 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.619359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.619371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.619422 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.620119 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.620619 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.621631 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.621834 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.621892 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.623998 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.623999 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.624000 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.624006 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.625340 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.625754 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.625755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.625756 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.625824 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.626083 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.626084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.626084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.626229 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.626299 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.626592 (+ 293us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.626682 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:07.626910 9010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.618118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42161 (local address 127.8.13.193:50453)
0925 14:58:07.618289 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.618326 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.619239 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.619505 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.619508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.619804 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:07.619998 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.620003 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.620708 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.620709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.621520 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.621522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.621910 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.621911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.622038 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.622510 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:07.622519 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:07.623909 (+ 1390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:07.625429 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.625436 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.625438 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.625656 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.625904 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.625905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.625905 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.626017 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.626303 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:07.626310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:07.626451 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.626563 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.626703 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":81,"threads_started":1}
I20260925 14:58:07.627429 9008 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42161
I20260925 14:58:07.627516 9008 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:07.627713 9008 heartbeater.cc:511] Master 127.8.13.254:42161 requested a full tablet report, sending...
I20260925 14:58:07.628284 8817 ts_manager.cc:194] Registered new tserver with Master: 95420da8fd994e318c7f6bb14a6a5289 (127.8.13.193:41729)
I20260925 14:58:07.628902 8817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:50453
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:07.650491 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.729121 9013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.729322 9013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.729346 9013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.730481 9013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.730516 9013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.730540 9013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:07.731966 9013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:07.732154 9013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.732331 9013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.737000 9021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.737025 9018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.737107 9013 server_base.cc:1034] running on GCE node
W20260925 14:58:07.737031 9019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.737429 9013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.737641 9013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.738860 9013 hybrid_clock.cc:648] HybridClock initialized: now 1790348287738848 us; error 29 us; skew 500 ppm
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.741534 9013 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:07.742040 9013 webserver.cc:460] Webserver started at http://127.8.13.194:44597/ using document root <none> and password file <none>
I20260925 14:58:07.742254 9013 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.742301 9013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.742426 9013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.743448 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "8e33bc66e70045469e7e8aea709715ed"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "9562542859c65ba88a81e6f8581dfd9e"
server_key_iv: "e779a6f25d46d5f02768b3f61ec3d4eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.743818 9013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "8e33bc66e70045469e7e8aea709715ed"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "9562542859c65ba88a81e6f8581dfd9e"
server_key_iv: "e779a6f25d46d5f02768b3f61ec3d4eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.745083 9013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:07.745828 9028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.745986 9013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:07.746078 9013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "8e33bc66e70045469e7e8aea709715ed"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "9562542859c65ba88a81e6f8581dfd9e"
server_key_iv: "e779a6f25d46d5f02768b3f61ec3d4eb"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.746150 9013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.765908 9013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.768131 9013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.768270 9013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.768496 9013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:07.768795 9013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:07.768827 9013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.768857 9013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:07.768874 9013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.774156 9013 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45787
I20260925 14:58:07.774204 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45787 every 8 connection(s)
I20260925 14:58:07.774506 9013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:07.775565 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9013
I20260925 14:58:07.775660 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:07.775822 8247 external_mini_cluster.cc:1428] Setting key bf487e0273ec7182a0abccd27237d7b4
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:07.784762 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.775279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:43973 (local address 127.8.13.254:42161)
0925 14:58:07.775341 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.775343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.775615 (+ 272us) server_negotiation.cc:407] Connection header received
0925 14:58:07.776760 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.776761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.776771 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.776816 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.777581 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.778061 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.779352 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.779670 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.779739 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.782022 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.782024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.782024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.782032 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.783493 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.783907 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.783908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.783908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.783970 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.784183 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.784184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.784184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.784298 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.784351 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.784577 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.784641 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:07.784945 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.775262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42161 (local address 127.8.13.194:43973)
0925 14:58:07.775447 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.775495 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.776655 (+ 1160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.776897 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.776901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.777229 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:07.777494 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.777509 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.778145 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.778146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.779219 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.779222 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.779742 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.779743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.779849 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.780450 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:07.780458 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:07.781929 (+ 1471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:07.783631 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.783637 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.783638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.783835 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.784028 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.784028 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.784029 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.784124 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.784349 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:07.784352 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:07.784460 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.784540 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.784691 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260925 14:58:07.785413 9142 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42161
I20260925 14:58:07.785494 9142 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:07.785655 9142 heartbeater.cc:511] Master 127.8.13.254:42161 requested a full tablet report, sending...
I20260925 14:58:07.786063 8817 ts_manager.cc:194] Registered new tserver with Master: 8e33bc66e70045469e7e8aea709715ed (127.8.13.194:45787)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
I20260925 14:58:07.786459 8817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:43973
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:07.799553 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:07.879767 9147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:07.879979 9147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:07.880007 9147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:07.881194 9147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:07.881228 9147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:07.881253 9147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:07.882655 9147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42161
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:07.882828 9147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:07.883009 9147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:07.887738 9153 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:07.887786 9155 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.887818 9147 server_base.cc:1034] running on GCE node
W20260925 14:58:07.887754 9152 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:07.888118 9147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:07.888371 9147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:07.889518 9147 hybrid_clock.cc:648] HybridClock initialized: now 1790348287889497 us; error 34 us; skew 500 ppm
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:07.892192 9147 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:07.892593 9147 webserver.cc:460] Webserver started at http://127.8.13.195:44585/ using document root <none> and password file <none>
I20260925 14:58:07.892776 9147 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:07.892813 9147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:07.892900 9147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:07.893859 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "006f5b315d0244ec84440a72b3013aae"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "534ad3ff6355351555218bded4e99b72"
server_key_iv: "9aca3c4ecd5bc774928d7c3e63c0d19d"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.894145 9147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "006f5b315d0244ec84440a72b3013aae"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "534ad3ff6355351555218bded4e99b72"
server_key_iv: "9aca3c4ecd5bc774928d7c3e63c0d19d"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.895319 9147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:07.896091 9162 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.896225 9147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.896302 9147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "006f5b315d0244ec84440a72b3013aae"
format_stamp: "Formatted at 2026-09-25 14:58:07 on dist-test-slave-15hz"
server_key: "534ad3ff6355351555218bded4e99b72"
server_key_iv: "9aca3c4ecd5bc774928d7c3e63c0d19d"
server_key_version: "encryptionkey@0"
I20260925 14:58:07.896368 9147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:07.905784 9147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:07.907356 9147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:07.907469 9147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:07.907732 9147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:07.907997 9147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:07.908036 9147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.908057 9147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:07.908080 9147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:07.913484 9147 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:45851
I20260925 14:58:07.913537 9275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:45851 every 8 connection(s)
I20260925 14:58:07.913798 9147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:07 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:07.922042 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.914520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:48207 (local address 127.8.13.254:42161)
0925 14:58:07.914578 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.914580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.914741 (+ 161us) server_negotiation.cc:407] Connection header received
0925 14:58:07.915633 (+ 892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.915634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.915644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.915695 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.916226 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.916706 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.917656 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.917830 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.917873 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.919788 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.919790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.919790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.919796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.920967 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.921294 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.921294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.921295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.921339 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.921580 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.921581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.921581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.921672 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.921715 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.921922 (+ 207us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.921978 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:58:07.922225 9278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.914477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42161 (local address 127.8.13.195:48207)
0925 14:58:07.914625 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.914664 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.915511 (+ 847us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.915750 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.915753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.915978 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:07.916167 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.916171 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.916769 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.916770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.917593 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.917595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.917891 (+ 296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.917892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.917984 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.918409 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:07.918417 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:07.919711 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:07.921041 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.921046 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.921048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.921239 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.921413 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:07.921414 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:07.921415 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:07.921524 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:07.921715 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:07.921718 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:07.921826 (+ 108us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.921902 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.922026 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260925 14:58:07.922658 9276 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42161
I20260925 14:58:07.922739 9276 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:07.922904 9276 heartbeater.cc:511] Master 127.8.13.254:42161 requested a full tablet report, sending...
I20260925 14:58:07.923257 8817 ts_manager.cc:194] Registered new tserver with Master: 006f5b315d0244ec84440a72b3013aae (127.8.13.195:45851)
I20260925 14:58:07.923511 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9147
I20260925 14:58:07.923630 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:07.923728 8817 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:48207
I20260925 14:58:07.923779 8247 external_mini_cluster.cc:1428] Setting key 7960f9d5497f1f3f7f0ba1f4fec3b158
I20260925 14:58:07.924507 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:07.930922 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.925974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.8.13.254:42161)
0925 14:58:07.926012 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.926014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.926044 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:07.926101 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.926102 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.926110 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.926160 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:07.926655 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.927110 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.927822 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.928017 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.928076 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.929010 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:07.929011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:07.929011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:07.929018 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:07.929943 (+ 925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.930233 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.930234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.930235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.930285 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:07.930443 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:07.930443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:07.930444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:07.930578 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:07.930660 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.930775 (+ 115us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.930840 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:07.932880 8817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:07.934552 8817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:07.940670 9289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.936935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49490 (local address 127.8.13.194:45787)
0925 14:58:07.937089 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.937094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.937580 (+ 486us) server_negotiation.cc:407] Connection header received
0925 14:58:07.937623 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.937626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.937641 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.937707 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:07.937897 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.938616 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.939614 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.940278 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940342 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.940404 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.940546 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.940585 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260925 14:58:07.940662 9288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.936825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45787 (local address 127.0.0.1:49490)
0925 14:58:07.937125 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.937166 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.937236 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.937789 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.937790 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.937793 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:07.937828 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.937829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.938718 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.938719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.939506 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.939509 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.940359 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.940442 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.940455 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.940520 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.940585 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":40,"threads_started":1}
I20260925 14:58:07.941099 9287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.937032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:45851 (local address 127.0.0.1:47458)
0925 14:58:07.937216 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.937225 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.937262 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.937516 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.937517 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.937519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:07.937624 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.937626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.938579 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.938580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.939986 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.939988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.940873 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.940933 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.940938 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.940981 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.941019 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":23,"thread_start_us":66,"threads_started":1}
I20260925 14:58:07.941236 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.937234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47458 (local address 127.8.13.195:45851)
0925 14:58:07.937373 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.937377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.937387 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:07.937420 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.937422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.937437 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.937506 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:07.937690 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.938483 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940077 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.940792 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940857 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.940910 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.940994 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.941153 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 14:58:07.941938 9291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.936825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41729 (local address 127.0.0.1:51678)
0925 14:58:07.937134 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:07.937167 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:07.937231 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:07.937510 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:07.937513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:07.937515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:07.937614 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.937616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.939835 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.939836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.940879 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:07.940881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.941710 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.941711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:07.941771 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.941774 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:07.941827 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:58:07.941865 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":6,"thread_start_us":87,"threads_started":1}
I20260925 14:58:07.941947 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:07.936942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51678 (local address 127.8.13.193:41729)
0925 14:58:07.937366 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:07.937369 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:07.937377 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:07.937409 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:07.937410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:07.937422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:07.937487 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:07.938609 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.939734 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.940966 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:07.941629 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:07.941688 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:07.941742 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:07.941840 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 14:58:07.941876 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":68,"threads_started":1}
I20260925 14:58:07.941890 9076 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1a24b43d834874acdf7d4a0eca0c9e (DEFAULT_TABLE table=test-table [id=3cf008745896442aa70dd6950fbad539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:07.941890 9210 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1a24b43d834874acdf7d4a0eca0c9e (DEFAULT_TABLE table=test-table [id=3cf008745896442aa70dd6950fbad539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:07.942258 9076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1a24b43d834874acdf7d4a0eca0c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:07.942258 9210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1a24b43d834874acdf7d4a0eca0c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:07.942526 8942 tablet_service.cc:1467] Processing CreateTablet for tablet 4e1a24b43d834874acdf7d4a0eca0c9e (DEFAULT_TABLE table=test-table [id=3cf008745896442aa70dd6950fbad539]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:07.942759 8942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e1a24b43d834874acdf7d4a0eca0c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:07.945233 9293 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: Bootstrap starting.
I20260925 14:58:07.945847 9293 tablet_bootstrap.cc:654] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:07.946113 9294 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: Bootstrap starting.
I20260925 14:58:07.946350 9293 log.cc:824] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:07.946708 9294 tablet_bootstrap.cc:654] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:07.946985 9293 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: No bootstrap required, opened a new log
I20260925 14:58:07.947043 9293 ts_tablet_manager.cc:1397] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:07.947150 9294 log.cc:824] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:07.947424 9295 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Bootstrap starting.
I20260925 14:58:07.947847 9294 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: No bootstrap required, opened a new log
I20260925 14:58:07.947918 9294 ts_tablet_manager.cc:1397] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:07.947957 9295 tablet_bootstrap.cc:654] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:07.948379 9295 log.cc:824] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:07.949093 9295 tablet_bootstrap.cc:492] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: No bootstrap required, opened a new log
I20260925 14:58:07.949153 9295 ts_tablet_manager.cc:1397] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:07.949018 9293 raft_consensus.cc:348] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.949193 9293 raft_consensus.cc:374] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:07.949235 9293 raft_consensus.cc:729] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95420da8fd994e318c7f6bb14a6a5289, State: Initialized, Role: FOLLOWER
I20260925 14:58:07.949350 9293 consensus_queue.cc:260] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.949625 9293 ts_tablet_manager.cc:1428] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:07.949783 9008 heartbeater.cc:503] Master 127.8.13.254:42161 was elected leader, sending a full tablet report...
I20260925 14:58:07.949942 9294 raft_consensus.cc:348] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.950127 9294 raft_consensus.cc:374] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:07.950169 9294 raft_consensus.cc:729] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 006f5b315d0244ec84440a72b3013aae, State: Initialized, Role: FOLLOWER
I20260925 14:58:07.950276 9294 consensus_queue.cc:260] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.950521 9276 heartbeater.cc:503] Master 127.8.13.254:42161 was elected leader, sending a full tablet report...
I20260925 14:58:07.950526 9294 ts_tablet_manager.cc:1428] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:07.951069 9295 raft_consensus.cc:348] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.951236 9295 raft_consensus.cc:374] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:07.951282 9295 raft_consensus.cc:729] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e33bc66e70045469e7e8aea709715ed, State: Initialized, Role: FOLLOWER
I20260925 14:58:07.951385 9295 consensus_queue.cc:260] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:07.951678 9295 ts_tablet_manager.cc:1428] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260925 14:58:07.951696 9142 heartbeater.cc:503] Master 127.8.13.254:42161 was elected leader, sending a full tablet report...
I20260925 14:58:08.018805 9301 raft_consensus.cc:482] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:08.018931 9301 raft_consensus.cc:504] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:08.019284 9301 leader_election.cc:288] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95420da8fd994e318c7f6bb14a6a5289 (127.8.13.193:41729), 006f5b315d0244ec84440a72b3013aae (127.8.13.195:45851)
I20260925 14:58:08.023343 9292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.019810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:37767 (local address 127.8.13.195:45851)
0925 14:58:08.019856 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.019858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.019954 (+ 96us) server_negotiation.cc:407] Connection header received
0925 14:58:08.019993 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.019994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.020005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.020062 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:08.020167 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.020831 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.022356 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.022932 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.023008 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.023017 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.023230 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.023270 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:08.023343 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.019664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:49929 (local address 127.8.13.193:41729)
0925 14:58:08.019722 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.019724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.019735 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:08.019767 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.019768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.019780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.019850 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:08.020038 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.020799 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.022356 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.022932 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.023008 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.023017 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.023229 (+ 212us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.023271 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:58:08.023353 9144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.019557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41729 (local address 127.8.13.194:49929)
0925 14:58:08.019616 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:08.019630 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:08.019679 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:08.019886 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:08.019887 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:08.019890 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:08.019966 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:08.019968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.020940 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.020941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:08.022243 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:08.022247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.023039 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.023040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:08.023157 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.023159 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:08.023209 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:58:08.023255 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":41}
I20260925 14:58:08.023360 9302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.019706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:45851 (local address 127.8.13.194:37767)
0925 14:58:08.019886 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:08.019894 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:08.019947 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:08.020075 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:08.020076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:08.020077 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:08.020123 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:08.020124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.020940 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.020941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:08.022243 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:08.022247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.023039 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.023040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:08.023141 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.023146 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:08.023209 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:08.023255 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":61,"thread_start_us":123,"threads_started":1}
I20260925 14:58:08.023737 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1a24b43d834874acdf7d4a0eca0c9e" candidate_uuid: "8e33bc66e70045469e7e8aea709715ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95420da8fd994e318c7f6bb14a6a5289" is_pre_election: true
I20260925 14:58:08.023871 8962 raft_consensus.cc:2454] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e33bc66e70045469e7e8aea709715ed in term 0.
I20260925 14:58:08.024068 9031 leader_election.cc:302] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8e33bc66e70045469e7e8aea709715ed, 95420da8fd994e318c7f6bb14a6a5289; no voters:
I20260925 14:58:08.024197 9301 raft_consensus.cc:2781] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:08.024192 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1a24b43d834874acdf7d4a0eca0c9e" candidate_uuid: "8e33bc66e70045469e7e8aea709715ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006f5b315d0244ec84440a72b3013aae" is_pre_election: true
I20260925 14:58:08.024273 9301 raft_consensus.cc:482] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:08.024296 9301 raft_consensus.cc:3037] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:08.024333 9230 raft_consensus.cc:2454] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e33bc66e70045469e7e8aea709715ed in term 0.
W20260925 14:58:08.025041 9143 tablet.cc:2369] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:08.025130 9301 raft_consensus.cc:504] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:08.025252 9301 leader_election.cc:288] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [CANDIDATE]: Term 1 election: Requested vote from peers 95420da8fd994e318c7f6bb14a6a5289 (127.8.13.193:41729), 006f5b315d0244ec84440a72b3013aae (127.8.13.195:45851)
I20260925 14:58:08.025386 8962 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1a24b43d834874acdf7d4a0eca0c9e" candidate_uuid: "8e33bc66e70045469e7e8aea709715ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95420da8fd994e318c7f6bb14a6a5289"
I20260925 14:58:08.025396 9230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e1a24b43d834874acdf7d4a0eca0c9e" candidate_uuid: "8e33bc66e70045469e7e8aea709715ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "006f5b315d0244ec84440a72b3013aae"
I20260925 14:58:08.025460 9230 raft_consensus.cc:3037] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:08.025462 8962 raft_consensus.cc:3037] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:08.026224 9230 raft_consensus.cc:2454] T 4e1a24b43d834874acdf7d4a0eca0c9e P 006f5b315d0244ec84440a72b3013aae [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e33bc66e70045469e7e8aea709715ed in term 1.
I20260925 14:58:08.026226 8962 raft_consensus.cc:2454] T 4e1a24b43d834874acdf7d4a0eca0c9e P 95420da8fd994e318c7f6bb14a6a5289 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e33bc66e70045469e7e8aea709715ed in term 1.
I20260925 14:58:08.026371 9031 leader_election.cc:302] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 006f5b315d0244ec84440a72b3013aae, 8e33bc66e70045469e7e8aea709715ed; no voters:
I20260925 14:58:08.026463 9301 raft_consensus.cc:2781] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:08.026604 9301 raft_consensus.cc:686] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [term 1 LEADER]: Becoming Leader. State: Replica: 8e33bc66e70045469e7e8aea709715ed, State: Running, Role: LEADER
I20260925 14:58:08.026681 9301 consensus_queue.cc:237] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } }
I20260925 14:58:08.027575 8816 catalog_manager.cc:5495] T 4e1a24b43d834874acdf7d4a0eca0c9e P 8e33bc66e70045469e7e8aea709715ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e33bc66e70045469e7e8aea709715ed (127.8.13.194). New cstate: current_term: 1 leader_uuid: "8e33bc66e70045469e7e8aea709715ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95420da8fd994e318c7f6bb14a6a5289" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e33bc66e70045469e7e8aea709715ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "006f5b315d0244ec84440a72b3013aae" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 45851 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348287, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:08.040522 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.033430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51692 (local address 127.8.13.193:41729)
0925 14:58:08.033492 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.033494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.033546 (+ 52us) server_negotiation.cc:407] Connection header received
0925 14:58:08.033620 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.033621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.033633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.033686 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.034204 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.034672 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.035390 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.035633 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.035690 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.037855 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.037869 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.037869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.037876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.039378 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.039753 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.039756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.039757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.039818 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.039987 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.039987 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.039987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.040138 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.040279 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.040342 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.040424 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:08.062098 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.055515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51706 (local address 127.8.13.193:41729)
0925 14:58:08.055597 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.055599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.055610 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:08.055691 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.055692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.055703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.055749 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.056330 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.056836 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.057556 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.057754 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.057809 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.059943 (+ 2134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.059945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.059945 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.059952 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.061155 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.061463 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.061464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.061464 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.061511 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.061647 (+ 136us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.061647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.061647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.061744 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.061808 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.061942 (+ 134us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.062002 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[8772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[8772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348288, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:08.082870 9290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:08.076316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51714 (local address 127.8.13.193:41729)
0925 14:58:08.076394 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:08.076396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:08.076444 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:58:08.076504 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:08.076506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:08.076514 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:08.076560 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:08.077056 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.077514 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.078229 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:08.078399 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:08.078441 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:08.080661 (+ 2220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:08.080663 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:08.080663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:08.080670 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:08.081869 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.082187 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.082187 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.082188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.082240 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:08.082463 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:08.082464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:08.082465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:08.082576 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:08.082669 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:08.082741 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 14:58:08.082803 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260925 14:58:08.083314 8922 tablet_service.cc:2908] Rejecting scan request for tablet 4e1a24b43d834874acdf7d4a0eca0c9e: Uninitialized: clean time has not yet been initialized
I20260925 14:58:08.084192 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8879
I20260925 14:58:08.089006 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9013
I20260925 14:58:08.092727 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9147
I20260925 14:58:08.096231 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 8785
2026-09-25T14:58:08Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1890 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[9317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[9317](info): set up 2 sockets
Sep 25 14:58:08 dist-test-slave-15hz krb5kdc[9317](info): commencing operation
krb5kdc: starting...
W20260925 14:58:09.046895 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:09.158275 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43557
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:43557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:09.237766 9330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:09.237964 9330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:09.237994 9330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:09.239125 9330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:09.239164 9330 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:58:09.239176 9330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:09.239189 9330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:09.239213 9330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:09.239234 9330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:09.240726 9330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:43557
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:09.240952 9330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:09.241164 9330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:09.245733 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.245757 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.245733 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:09.245826 9330 server_base.cc:1034] running on GCE node
I20260925 14:58:09.246206 9330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:09.246480 9330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:09.247602 9330 hybrid_clock.cc:648] HybridClock initialized: now 1790348289247588 us; error 28 us; skew 500 ppm
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:09.255643 9330 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:09.256218 9330 webserver.cc:460] Webserver started at http://127.8.13.254:38617/ using document root <none> and password file <none>
I20260925 14:58:09.256430 9330 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:09.256475 9330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:09.256608 9330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:09.257710 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "bf1144fb821947758f293561b9d23716"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "086b1b00462cd6a0fd95abef1a2d24b3"
server_key_iv: "8a845d448b4ae9bf5daa572317dcd3e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.258059 9330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "bf1144fb821947758f293561b9d23716"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "086b1b00462cd6a0fd95abef1a2d24b3"
server_key_iv: "8a845d448b4ae9bf5daa572317dcd3e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.259284 9330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:09.259994 9345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.260131 9330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.260212 9330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "bf1144fb821947758f293561b9d23716"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "086b1b00462cd6a0fd95abef1a2d24b3"
server_key_iv: "8a845d448b4ae9bf5daa572317dcd3e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.260280 9330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:09.274120 9330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:09.275902 9330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:09.276033 9330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:09.280411 9330 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:43557
I20260925 14:58:09.280457 9407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:43557 every 8 connection(s)
I20260925 14:58:09.280762 9330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:09.281389 9408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:09.282878 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9330
I20260925 14:58:09.282956 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:09.283108 8247 external_mini_cluster.cc:1428] Setting key 2241312a6c06fc8ad7bf81c530070e99
I20260925 14:58:09.283890 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Bootstrap starting.
I20260925 14:58:09.284451 9408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:09.284760 9408 log.cc:824] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:09.285480 9408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: No bootstrap required, opened a new log
I20260925 14:58:09.287608 9408 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } }
I20260925 14:58:09.287765 9408 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:09.287801 9408 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf1144fb821947758f293561b9d23716, State: Initialized, Role: FOLLOWER
I20260925 14:58:09.287971 9408 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } }
I20260925 14:58:09.288065 9408 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:09.288092 9408 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:09.288139 9408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:09.288645 9408 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } }
I20260925 14:58:09.288738 9408 leader_election.cc:302] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf1144fb821947758f293561b9d23716; no voters:
I20260925 14:58:09.288889 9408 leader_election.cc:288] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:09.288949 9413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:09.289147 9413 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [term 1 LEADER]: Becoming Leader. State: Replica: bf1144fb821947758f293561b9d23716, State: Running, Role: LEADER
I20260925 14:58:09.289184 9408 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:09.289304 9413 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } }
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:09.290426 9414 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf1144fb821947758f293561b9d23716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } } }
I20260925 14:58:09.290504 9414 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:09.291055 9428 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:09.291146 9428 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:09.291275 9429 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:09.291500 9429 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:09.291245 9415 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf1144fb821947758f293561b9d23716. Latest consensus state: current_term: 1 leader_uuid: "bf1144fb821947758f293561b9d23716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf1144fb821947758f293561b9d23716" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43557 } } }
I20260925 14:58:09.292272 9415 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:09.293197 9429 catalog_manager.cc:1347] Generated new cluster ID: 77a870f6957a4cccac6f256a6778ab62
I20260925 14:58:09.293248 9429 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:09.295063 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.283781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39422 (local address 127.8.13.254:43557)
0925 14:58:09.283916 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.283924 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.283949 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:58:09.284412 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.284419 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.284467 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.284630 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.285300 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.286204 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.287250 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.287686 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.287767 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.291102 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.291106 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.291107 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.291119 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.293401 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.293768 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.293771 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.293772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.293863 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.294072 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.294072 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.294073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.294313 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.294502 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.294569 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.294698 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":65,"threads_started":1}
I20260925 14:58:09.311890 9429 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:09.312637 9429 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:09.318450 9429 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Generated new TSK 0
I20260925 14:58:09.318677 9429 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:09.352231 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43557
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:09.455276 9434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:09.455487 9434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:09.455511 9434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:09.456724 9434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:09.456765 9434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:09.456785 9434 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:09.456802 9434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:09.458197 9434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:09.458388 9434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:09.458568 9434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:09.463267 9442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.463305 9439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.463279 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:09.463472 9434 server_base.cc:1034] running on GCE node
I20260925 14:58:09.463721 9434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:09.463981 9434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:09.465133 9434 hybrid_clock.cc:648] HybridClock initialized: now 1790348289465117 us; error 29 us; skew 500 ppm
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:09.467429 9434 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:09.467916 9434 webserver.cc:460] Webserver started at http://127.8.13.193:37791/ using document root <none> and password file <none>
I20260925 14:58:09.468106 9434 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:09.468143 9434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:09.468235 9434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:09.469234 9434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "83c405c145624b69991819708616e89b"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "2c53be52c537aa0c8307f00eb3ca905c"
server_key_iv: "49c0784c4868796ccc4cac3a4d4fe697"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.469515 9434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "83c405c145624b69991819708616e89b"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "2c53be52c537aa0c8307f00eb3ca905c"
server_key_iv: "49c0784c4868796ccc4cac3a4d4fe697"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.470671 9434 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:09.471276 9449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.471443 9434 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:09.471517 9434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "83c405c145624b69991819708616e89b"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "2c53be52c537aa0c8307f00eb3ca905c"
server_key_iv: "49c0784c4868796ccc4cac3a4d4fe697"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.471608 9434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:09.499993 9434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:09.501919 9434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:09.502033 9434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:09.502501 9434 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:09.502535 9434 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.502561 9434 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:09.502574 9434 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.508937 9434 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:33997
I20260925 14:58:09.509004 9564 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:33997 every 8 connection(s)
I20260925 14:58:09.509321 9434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:09.512228 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.503106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:36805 (local address 127.8.13.254:43557)
0925 14:58:09.503158 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.503160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.503405 (+ 245us) server_negotiation.cc:407] Connection header received
0925 14:58:09.503931 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.503932 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.503946 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.504023 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.504855 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.505616 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.507125 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.507440 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.507563 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.509962 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.509963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.509963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.509970 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.511014 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.511365 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.511365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.511366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.511408 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.511625 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.511625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.511626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.511771 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.511836 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.512095 (+ 259us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.512162 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:09.512398 9458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.503111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.193:36805)
0925 14:58:09.503268 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.503305 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.503816 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.504100 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.504105 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.504441 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.504743 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.504750 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.505707 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.505708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.507041 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.507044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.507561 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.507562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.507702 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.508226 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.508236 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.509878 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.511086 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.511092 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.511093 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.511310 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.511455 (+ 145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.511455 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.511456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.511574 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.511849 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.511855 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.511984 (+ 129us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.512065 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.512184 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
I20260925 14:58:09.512856 9565 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43557
I20260925 14:58:09.512934 9565 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:09.513108 9565 heartbeater.cc:511] Master 127.8.13.254:43557 requested a full tablet report, sending...
I20260925 14:58:09.513576 9361 ts_manager.cc:194] Registered new tserver with Master: 83c405c145624b69991819708616e89b (127.8.13.193:33997)
I20260925 14:58:09.514364 9361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:36805
I20260925 14:58:09.517866 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9434
I20260925 14:58:09.517976 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:09.518157 8247 external_mini_cluster.cc:1428] Setting key 06799478ef1d8026a92dda2499e0ba76
I20260925 14:58:09.518822 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.513919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:53911 (local address 127.8.13.254:43557)
0925 14:58:09.513953 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.513955 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.514080 (+ 125us) server_negotiation.cc:407] Connection header received
0925 14:58:09.514102 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.514102 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.514113 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.514168 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.514431 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.514984 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.515698 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.515876 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.515923 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.516714 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.516715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.516715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.516721 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.517755 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.518111 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.518112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.518112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.518167 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.518362 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.518363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.518363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.518485 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.518538 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.518689 (+ 151us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.518748 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:58:09.518868 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.513857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.193:53911)
0925 14:58:09.514010 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.514019 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.514069 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.514172 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.514172 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.514317 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.514368 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.514370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.515078 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.515079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.515636 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.515638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.515930 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.515930 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.515980 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.516374 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.516379 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.516635 (+ 256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.517834 (+ 1199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.517835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.517835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.518038 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.518238 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.518238 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.518239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.518303 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.518553 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.518554 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.518601 (+ 47us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.518672 (+ 71us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.518777 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:09.546171 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43557
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:09.626892 9574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:09.627184 9574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:09.627223 9574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:09.628427 9574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:09.628474 9574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:09.628505 9574 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:09.628531 9574 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:09.630009 9574 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:09.630254 9574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:09.630517 9574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:09.635391 9580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.635386 9582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:09.635462 9574 server_base.cc:1034] running on GCE node
W20260925 14:58:09.635386 9579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:09.635812 9574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:09.636058 9574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:09.637210 9574 hybrid_clock.cc:648] HybridClock initialized: now 1790348289637197 us; error 29 us; skew 500 ppm
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:09.639838 9574 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:09.640416 9574 webserver.cc:460] Webserver started at http://127.8.13.194:36153/ using document root <none> and password file <none>
I20260925 14:58:09.640650 9574 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:09.640702 9574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:09.640825 9574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:09.641880 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "6cf9a94715b14866be80f85e02efd845"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "78c7380705ac7082460e8f5520ef659a"
server_key_iv: "7ad3c96c1bcfdb10010bebf8ac1b375f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.642208 9574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "6cf9a94715b14866be80f85e02efd845"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "78c7380705ac7082460e8f5520ef659a"
server_key_iv: "7ad3c96c1bcfdb10010bebf8ac1b375f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.643503 9574 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:09.644281 9589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.644456 9574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:09.644544 9574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "6cf9a94715b14866be80f85e02efd845"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "78c7380705ac7082460e8f5520ef659a"
server_key_iv: "7ad3c96c1bcfdb10010bebf8ac1b375f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.644609 9574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:09.652456 9574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:09.654255 9574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:09.654395 9574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:09.654979 9574 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:09.655031 9574 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.655061 9574 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:09.655076 9574 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:09.661885 9704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:41815 every 8 connection(s)
I20260925 14:58:09.661835 9574 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:41815
I20260925 14:58:09.662807 9574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:09.665350 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.655479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:55213 (local address 127.8.13.254:43557)
0925 14:58:09.655566 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.655569 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.655771 (+ 202us) server_negotiation.cc:407] Connection header received
0925 14:58:09.656249 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.656251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.656265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.656334 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.657018 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.657796 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.659364 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.659641 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.659700 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.662308 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.662310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.662310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.662320 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.663942 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.664369 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.664371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.664371 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.664437 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.664666 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.664668 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.664668 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.664808 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.664865 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.665162 (+ 297us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.665259 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:09.665535 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.655440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.194:55213)
0925 14:58:09.655638 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.655681 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.656156 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.656413 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.656417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.656700 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.656921 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.656927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.657899 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.657900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.659264 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.659267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.659723 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.659724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.659899 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.660433 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.660443 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.662154 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.664034 (+ 1880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.664041 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.664044 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.664291 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.664498 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.664499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.664499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.664606 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.664870 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.664877 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.665021 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.665127 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.665306 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":116,"threads_started":1}
I20260925 14:58:09.666014 9705 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43557
I20260925 14:58:09.666097 9705 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:09.666270 9705 heartbeater.cc:511] Master 127.8.13.254:43557 requested a full tablet report, sending...
I20260925 14:58:09.666627 9362 ts_manager.cc:194] Registered new tserver with Master: 6cf9a94715b14866be80f85e02efd845 (127.8.13.194:41815)
I20260925 14:58:09.667011 9362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:55213
I20260925 14:58:09.670990 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9574
I20260925 14:58:09.671104 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:09.671307 8247 external_mini_cluster.cc:1428] Setting key 52ed122d2f865aa86c24a57f0ac54fb0
I20260925 14:58:09.672698 9711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.667168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.194:56019)
0925 14:58:09.667295 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.667306 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.667373 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.667501 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.667501 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.667716 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.667776 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.667778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.668423 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.668423 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.668933 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.668934 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.669277 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.669278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.669333 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.669776 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.669781 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.670063 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.671663 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.671663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.671664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.671870 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.672094 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.672095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.672095 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.672156 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.672463 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.672463 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.672500 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.672543 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.672628 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":73,"threads_started":1}
I20260925 14:58:09.672730 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.667160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56019 (local address 127.8.13.254:43557)
0925 14:58:09.667206 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.667209 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.667369 (+ 160us) server_negotiation.cc:407] Connection header received
0925 14:58:09.667416 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.667417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.667428 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.667511 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.667849 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.668339 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.669024 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.669214 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.669275 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.670166 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.670167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.670168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.670175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.671577 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.671960 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.671961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.671961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.672029 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.672213 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.672214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.672214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.672406 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.672475 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.672569 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.672645 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:09.694459 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43557
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:09.773190 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:09.773442 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:09.773476 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:09.775098 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:09.775142 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:09.775169 9714 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:09.775192 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:09.776649 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:43557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:09.776883 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:09.777107 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:09.781709 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.781764 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:09.781771 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:09.781864 9714 server_base.cc:1034] running on GCE node
I20260925 14:58:09.782160 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:09.782383 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:09.783569 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790348289783543 us; error 32 us; skew 500 ppm
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:09.786005 9714 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:09.786504 9714 webserver.cc:460] Webserver started at http://127.8.13.195:40583/ using document root <none> and password file <none>
I20260925 14:58:09.786701 9714 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:09.786747 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:09.786873 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:09.787932 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "d9c807017ce24a4e8617cf28c054bbee"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "1b205c95cc644105d93886ed3166eb55"
server_key_iv: "b038d2e3e4db352ec17f63d79967903f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.788261 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "d9c807017ce24a4e8617cf28c054bbee"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "1b205c95cc644105d93886ed3166eb55"
server_key_iv: "b038d2e3e4db352ec17f63d79967903f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.789559 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:09.790361 9729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.790530 9714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:09.790616 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "d9c807017ce24a4e8617cf28c054bbee"
format_stamp: "Formatted at 2026-09-25 14:58:09 on dist-test-slave-15hz"
server_key: "1b205c95cc644105d93886ed3166eb55"
server_key_iv: "b038d2e3e4db352ec17f63d79967903f"
server_key_version: "encryptionkey@0"
I20260925 14:58:09.790697 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:09.803184 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:09.805413 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:09.805528 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:09.805987 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:09.806025 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.806066 9714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:09.806087 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:09.812093 9714 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:37793
I20260925 14:58:09.812386 9844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:37793 every 8 connection(s)
I20260925 14:58:09.812558 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:09.816148 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.806660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:60921 (local address 127.8.13.254:43557)
0925 14:58:09.806714 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.806716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.806905 (+ 189us) server_negotiation.cc:407] Connection header received
0925 14:58:09.807516 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.807518 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.807552 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.807626 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.808472 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.809229 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.810640 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.810930 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.811014 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.813592 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.813594 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.813594 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.813602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.814903 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.815259 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.815260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.815261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.815314 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.815549 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.815550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.815550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.815664 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.815720 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.815985 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.816060 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:58:09.816311 9738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.806631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.195:60921)
0925 14:58:09.806783 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.806820 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.807377 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.807707 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.807711 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.808070 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.808362 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.808369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.809319 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.809320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.810551 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.810554 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.811049 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.811050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.811195 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.811754 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.811762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.813502 (+ 1740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.814980 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.814986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.814988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.815192 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.815368 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.815369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.815369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.815473 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.815724 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.815730 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.815863 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.815951 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.816107 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260925 14:58:09.816767 9845 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43557
I20260925 14:58:09.816859 9845 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:09.817082 9845 heartbeater.cc:511] Master 127.8.13.254:43557 requested a full tablet report, sending...
I20260925 14:58:09.817449 9361 ts_manager.cc:194] Registered new tserver with Master: d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:09.817860 9361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:60921
I20260925 14:58:09.819097 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9714
I20260925 14:58:09.819164 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:09.819325 8247 external_mini_cluster.cc:1428] Setting key 310a76bfe64e6b2ff312acc71b4cc17f
I20260925 14:58:09.820132 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:09.823001 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.817759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:48257 (local address 127.8.13.254:43557)
0925 14:58:09.817796 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.817798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.817851 (+ 53us) server_negotiation.cc:407] Connection header received
0925 14:58:09.817899 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.817900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.817910 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.817964 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.818276 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.818847 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.819765 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.820004 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.820063 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.820844 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.820846 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.820846 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.820852 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.822057 (+ 1205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.822402 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.822403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.822403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.822443 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.822592 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.822593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.822593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.822673 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.822718 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.822866 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.822928 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:09.823004 9851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.817690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.8.13.195:48257)
0925 14:58:09.817801 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.817810 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.817863 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.817994 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.817995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.818143 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:09.818200 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.818202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.818932 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.818933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.819658 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.819661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.820071 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.820072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.820146 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.820510 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:09.820525 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:09.820768 (+ 243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:09.822139 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.822140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.822140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.822337 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.822480 (+ 143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:09.822481 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:09.822481 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:09.822544 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:09.822727 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:09.822728 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:09.822771 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.822835 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.822921 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"thread_start_us":62,"threads_started":1}
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:09 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:09.844457 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.838141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39432 (local address 127.8.13.254:43557)
0925 14:58:09.838202 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.838204 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.838213 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:09.838238 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.838239 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.838248 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.838294 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:09.838790 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.839308 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.840065 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.840253 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.840301 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.842251 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:09.842253 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:09.842253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:09.842260 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:09.843406 (+ 1146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.843766 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.843767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.843768 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.843820 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:09.843988 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:09.843989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:09.843989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:09.844099 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:09.844186 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.844289 (+ 103us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.844364 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:09.846190 9361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:09.847869 9361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:09.854300 9863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33997 (local address 127.0.0.1:43960)
0925 14:58:09.850341 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.850384 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.850444 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.850720 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.850723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.850727 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:09.850798 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.850801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.852211 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.852212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.853215 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.853217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.854069 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.854069 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.854126 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.854137 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.854186 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.854223 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":37,"thread_start_us":44,"threads_started":1}
I20260925 14:58:09.854353 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.8.13.193:33997)
0925 14:58:09.850540 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.850543 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.850552 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:09.850590 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.850592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.850609 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.850692 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:09.850916 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.852124 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.853293 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.853990 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.854069 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.854131 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.854215 (+ 84us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.854256 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":80,"threads_started":1}
I20260925 14:58:09.854745 9862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41815 (local address 127.0.0.1:55110)
0925 14:58:09.850468 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.850479 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.850536 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.850773 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.850774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.850775 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:09.850830 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.850832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.852197 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.852198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.853439 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.853441 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.854307 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.854308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.854363 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.854367 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.854635 (+ 268us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.854680 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"mutex_wait_us":49,"thread_start_us":89,"threads_started":1}
I20260925 14:58:09.854853 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55110 (local address 127.8.13.194:41815)
0925 14:58:09.850469 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.850474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.850599 (+ 125us) server_negotiation.cc:407] Connection header received
0925 14:58:09.850638 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.850640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.850653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.850710 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:09.850946 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.852107 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.853523 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.854226 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.854287 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.854338 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.854699 (+ 361us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.854750 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260925 14:58:09.855058 9499 tablet_service.cc:1467] Processing CreateTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad (DEFAULT_TABLE table=test-table [id=be7b7f1859d9474180d5efe374fc1516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:09.855417 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475c94d2e8a545f5ac02f42a9aed38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:09.856033 9866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:37793 (local address 127.0.0.1:50148)
0925 14:58:09.850780 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.850797 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.850838 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.851121 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.851122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.851124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:09.851170 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.851171 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.852072 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.852073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.853438 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.853441 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.855802 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.855803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.855862 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.855867 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.855928 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.855969 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"mutex_wait_us":26,"thread_start_us":71,"threads_started":1}
I20260925 14:58:09.856141 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.850639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50148 (local address 127.8.13.195:37793)
0925 14:58:09.850819 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.850823 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.850833 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:09.850957 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.850960 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.850976 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.851041 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:09.851242 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.851998 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.854731 (+ 2733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.855720 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.855784 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.855821 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.855988 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.856027 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":82,"threads_started":1}
I20260925 14:58:09.856734 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad (DEFAULT_TABLE table=test-table [id=be7b7f1859d9474180d5efe374fc1516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:09.857033 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475c94d2e8a545f5ac02f42a9aed38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:09.858619 9868 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Bootstrap starting.
I20260925 14:58:09.859304 9868 tablet_bootstrap.cc:654] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:09.859671 9868 log.cc:824] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:09.859932 9869 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Bootstrap starting.
I20260925 14:58:09.859961 9639 tablet_service.cc:1467] Processing CreateTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad (DEFAULT_TABLE table=test-table [id=be7b7f1859d9474180d5efe374fc1516]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:09.860353 9639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 475c94d2e8a545f5ac02f42a9aed38ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:09.860368 9868 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: No bootstrap required, opened a new log
I20260925 14:58:09.860426 9868 ts_tablet_manager.cc:1397] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:09.860960 9869 tablet_bootstrap.cc:654] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:09.861266 9869 log.cc:824] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:09.861909 9868 raft_consensus.cc:348] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.862039 9868 raft_consensus.cc:374] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:09.862082 9868 raft_consensus.cc:729] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c405c145624b69991819708616e89b, State: Initialized, Role: FOLLOWER
I20260925 14:58:09.862193 9868 consensus_queue.cc:260] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.862459 9868 ts_tablet_manager.cc:1428] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:09.862502 9565 heartbeater.cc:503] Master 127.8.13.254:43557 was elected leader, sending a full tablet report...
I20260925 14:58:09.863024 9869 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: No bootstrap required, opened a new log
I20260925 14:58:09.863083 9869 ts_tablet_manager.cc:1397] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:09.863801 9873 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Bootstrap starting.
I20260925 14:58:09.864336 9873 tablet_bootstrap.cc:654] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:09.864593 9873 log.cc:824] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:09.864876 9869 raft_consensus.cc:348] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.865003 9869 raft_consensus.cc:374] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:09.865034 9869 raft_consensus.cc:729] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c807017ce24a4e8617cf28c054bbee, State: Initialized, Role: FOLLOWER
I20260925 14:58:09.865144 9869 consensus_queue.cc:260] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.865239 9873 tablet_bootstrap.cc:492] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: No bootstrap required, opened a new log
I20260925 14:58:09.865306 9873 ts_tablet_manager.cc:1397] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:09.865398 9869 ts_tablet_manager.cc:1428] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:09.865427 9845 heartbeater.cc:503] Master 127.8.13.254:43557 was elected leader, sending a full tablet report...
I20260925 14:58:09.867090 9873 raft_consensus.cc:348] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.867218 9873 raft_consensus.cc:374] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:09.867249 9873 raft_consensus.cc:729] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf9a94715b14866be80f85e02efd845, State: Initialized, Role: FOLLOWER
I20260925 14:58:09.867357 9873 consensus_queue.cc:260] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.867669 9873 ts_tablet_manager.cc:1428] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:09.867666 9705 heartbeater.cc:503] Master 127.8.13.254:43557 was elected leader, sending a full tablet report...
W20260925 14:58:09.913522 9706 tablet.cc:2369] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:09.982856 9876 raft_consensus.cc:482] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:09.982980 9876 raft_consensus.cc:504] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.983309 9876 leader_election.cc:288] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c405c145624b69991819708616e89b (127.8.13.193:33997), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:09.986109 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.983650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:40781 (local address 127.8.13.195:37793)
0925 14:58:09.983711 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.983713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.983822 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:58:09.983871 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.983872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.983883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.983933 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:09.984075 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.984521 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985428 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.985785 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985844 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.985854 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.986003 (+ 149us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.986042 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 14:58:09.986101 9864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.983591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:36857 (local address 127.8.13.193:33997)
0925 14:58:09.983632 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.983634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.983733 (+ 99us) server_negotiation.cc:407] Connection header received
0925 14:58:09.983770 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.983771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.983786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.983837 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:09.984012 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.984503 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985427 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.985789 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985851 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.985859 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.985991 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.986030 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:09.986132 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.983562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33997 (local address 127.8.13.194:36857)
0925 14:58:09.983622 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.983632 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.983683 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.983865 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.983866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.983868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:09.983940 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.983942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.984576 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.984577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.985338 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.985340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.985860 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.985928 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.985932 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.985975 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.986021 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:58:09.986138 9877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.983567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:37793 (local address 127.8.13.194:40781)
0925 14:58:09.983753 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:09.983762 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:09.983820 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:09.983935 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:09.983935 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:09.983936 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:09.984006 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.984008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.984600 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.984601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.985340 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:09.985341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.985864 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.985865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:09.985938 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.985941 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:09.985988 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:09.986025 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":17,"thread_start_us":106,"threads_started":1}
I20260925 14:58:09.986501 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475c94d2e8a545f5ac02f42a9aed38ad" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c405c145624b69991819708616e89b" is_pre_election: true
I20260925 14:58:09.986666 9519 raft_consensus.cc:2454] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 0.
I20260925 14:58:09.986665 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475c94d2e8a545f5ac02f42a9aed38ad" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee" is_pre_election: true
I20260925 14:58:09.986783 9799 raft_consensus.cc:2454] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 0.
I20260925 14:58:09.986907 9592 leader_election.cc:302] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, 83c405c145624b69991819708616e89b; no voters:
I20260925 14:58:09.987046 9876 raft_consensus.cc:2781] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:09.987102 9876 raft_consensus.cc:482] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:09.987124 9876 raft_consensus.cc:3037] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:09.988099 9876 raft_consensus.cc:504] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.988225 9876 leader_election.cc:288] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 election: Requested vote from peers 83c405c145624b69991819708616e89b (127.8.13.193:33997), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:09.988381 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475c94d2e8a545f5ac02f42a9aed38ad" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c405c145624b69991819708616e89b"
I20260925 14:58:09.988412 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "475c94d2e8a545f5ac02f42a9aed38ad" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee"
I20260925 14:58:09.988488 9519 raft_consensus.cc:3037] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:09.988488 9799 raft_consensus.cc:3037] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:09.989431 9519 raft_consensus.cc:2454] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 1.
I20260925 14:58:09.989431 9799 raft_consensus.cc:2454] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 1.
I20260925 14:58:09.989640 9593 leader_election.cc:302] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, d9c807017ce24a4e8617cf28c054bbee; no voters:
I20260925 14:58:09.989732 9876 raft_consensus.cc:2781] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:09.989944 9876 raft_consensus.cc:686] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf9a94715b14866be80f85e02efd845, State: Running, Role: LEADER
I20260925 14:58:09.990056 9876 consensus_queue.cc:237] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:09.990972 9362 catalog_manager.cc:5495] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf9a94715b14866be80f85e02efd845 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:10.000038 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:09.997461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.8.13.194:41815)
0925 14:58:09.997522 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:09.997524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:09.997533 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:09.997636 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:09.997637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:09.997659 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:09.997719 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:09.997862 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.998351 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.999162 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:09.999363 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:09.999417 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:09.999673 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:09.999781 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:09.999851 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:09.999905 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:58:09.999930 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:10.001838 9519 raft_consensus.cc:1261] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Refusing update from remote peer 6cf9a94715b14866be80f85e02efd845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:10.001838 9799 raft_consensus.cc:1261] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Refusing update from remote peer 6cf9a94715b14866be80f85e02efd845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:10.002234 9878 consensus_queue.cc:1035] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.002360 9876 consensus_queue.cc:1035] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.008213 9361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:10.008303 9361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:10.009002 9361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Sending DeleteTablet for 3 replicas of tablet 475c94d2e8a545f5ac02f42a9aed38ad
I20260925 14:58:10.009305 9499 tablet_service.cc:1514] Processing DeleteTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:43960
I20260925 14:58:10.009387 9639 tablet_service.cc:1514] Processing DeleteTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260925 14:58:10.009423 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet 475c94d2e8a545f5ac02f42a9aed38ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50148
I20260925 14:58:10.009534 9890 tablet_replica.cc:330] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: stopping tablet replica
I20260925 14:58:10.009644 9890 raft_consensus.cc:2229] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:10.009750 9890 raft_consensus.cc:2258] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.009737 9361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:10.009879 9361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:10.010363 9890 ts_tablet_manager.cc:1905] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.011214 9639 tablet_service.cc:1467] Processing CreateTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 (DEFAULT_TABLE table=test-table [id=b82fbadb1c524b41903707bf9f43ff2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:10.011336 9639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68b04e5c0a248bfa206356b1a9d5e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.011338 9779 tablet_service.cc:1467] Processing CreateTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 (DEFAULT_TABLE table=test-table [id=b82fbadb1c524b41903707bf9f43ff2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:10.011444 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68b04e5c0a248bfa206356b1a9d5e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.012413 9890 ts_tablet_manager.cc:1918] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:10.012472 9890 log.cc:1196] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/475c94d2e8a545f5ac02f42a9aed38ad
I20260925 14:58:10.012692 9499 tablet_service.cc:1467] Processing CreateTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 (DEFAULT_TABLE table=test-table [id=b82fbadb1c524b41903707bf9f43ff2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:10.012719 9873 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Bootstrap starting.
I20260925 14:58:10.012784 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68b04e5c0a248bfa206356b1a9d5e82. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.013126 9873 tablet_bootstrap.cc:654] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.013197 9890 ts_tablet_manager.cc:1939] T 475c94d2e8a545f5ac02f42a9aed38ad P 83c405c145624b69991819708616e89b: Deleting consensus metadata
I20260925 14:58:10.013628 9348 catalog_manager.cc:4841] TS 83c405c145624b69991819708616e89b (127.8.13.193:33997): tablet 475c94d2e8a545f5ac02f42a9aed38ad (table test-table [id=be7b7f1859d9474180d5efe374fc1516]) successfully deleted
I20260925 14:58:10.013935 9868 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Bootstrap starting.
I20260925 14:58:10.014282 9868 tablet_bootstrap.cc:654] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.014333 9869 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Bootstrap starting.
I20260925 14:58:10.014842 9869 tablet_bootstrap.cc:654] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.014883 9868 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: No bootstrap required, opened a new log
I20260925 14:58:10.014918 9868 ts_tablet_manager.cc:1397] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:10.015062 9868 raft_consensus.cc:348] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.015136 9868 raft_consensus.cc:374] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.015156 9868 raft_consensus.cc:729] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c405c145624b69991819708616e89b, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.015201 9868 consensus_queue.cc:260] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.015282 9868 ts_tablet_manager.cc:1428] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:10.015581 9869 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: No bootstrap required, opened a new log
I20260925 14:58:10.015628 9869 ts_tablet_manager.cc:1397] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:10.015744 9869 raft_consensus.cc:348] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.015789 9869 raft_consensus.cc:374] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.015800 9869 raft_consensus.cc:729] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c807017ce24a4e8617cf28c054bbee, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.015832 9869 consensus_queue.cc:260] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.015906 9869 ts_tablet_manager.cc:1428] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:10.016152 9891 tablet_replica.cc:330] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: stopping tablet replica
I20260925 14:58:10.016278 9891 raft_consensus.cc:2229] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:10.016407 9891 raft_consensus.cc:2258] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.016582 9892 tablet_replica.cc:330] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: stopping tablet replica
I20260925 14:58:10.016714 9892 raft_consensus.cc:2229] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:10.016844 9892 raft_consensus.cc:2258] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.016878 9891 ts_tablet_manager.cc:1905] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.017230 9892 ts_tablet_manager.cc:1905] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.018379 9891 ts_tablet_manager.cc:1918] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:10.018446 9891 log.cc:1196] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/475c94d2e8a545f5ac02f42a9aed38ad
I20260925 14:58:10.018546 9892 ts_tablet_manager.cc:1918] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:10.018620 9892 log.cc:1196] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/475c94d2e8a545f5ac02f42a9aed38ad
I20260925 14:58:10.018672 9891 ts_tablet_manager.cc:1939] T 475c94d2e8a545f5ac02f42a9aed38ad P d9c807017ce24a4e8617cf28c054bbee: Deleting consensus metadata
I20260925 14:58:10.018848 9892 ts_tablet_manager.cc:1939] T 475c94d2e8a545f5ac02f42a9aed38ad P 6cf9a94715b14866be80f85e02efd845: Deleting consensus metadata
I20260925 14:58:10.019052 9349 catalog_manager.cc:4841] TS d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793): tablet 475c94d2e8a545f5ac02f42a9aed38ad (table test-table [id=be7b7f1859d9474180d5efe374fc1516]) successfully deleted
I20260925 14:58:10.019208 9349 catalog_manager.cc:4841] TS 6cf9a94715b14866be80f85e02efd845 (127.8.13.194:41815): tablet 475c94d2e8a545f5ac02f42a9aed38ad (table test-table [id=be7b7f1859d9474180d5efe374fc1516]) successfully deleted
I20260925 14:58:10.019887 9873 tablet_bootstrap.cc:492] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: No bootstrap required, opened a new log
I20260925 14:58:10.019938 9873 ts_tablet_manager.cc:1397] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260925 14:58:10.020079 9873 raft_consensus.cc:348] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.020148 9873 raft_consensus.cc:374] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.020167 9873 raft_consensus.cc:729] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf9a94715b14866be80f85e02efd845, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.020212 9873 consensus_queue.cc:260] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.020298 9873 ts_tablet_manager.cc:1428] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260925 14:58:10.063141 9846 tablet.cc:2369] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:10.190148 9872 raft_consensus.cc:482] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:10.190248 9872 raft_consensus.cc:504] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.190568 9872 leader_election.cc:288] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cf9a94715b14866be80f85e02efd845 (127.8.13.194:41815), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:10.193439 9865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.190895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:50653 (local address 127.8.13.194:41815)
0925 14:58:10.190956 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.190958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.191038 (+ 80us) server_negotiation.cc:407] Connection header received
0925 14:58:10.191121 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.191123 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.191133 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.191191 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:10.191328 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.191834 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.192755 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.193107 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.193171 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.193179 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.193332 (+ 153us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.193371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:58:10.193449 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.190806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41815 (local address 127.8.13.193:50653)
0925 14:58:10.190935 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.190948 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.191045 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.191197 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.191199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.191202 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:10.191257 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.191259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.191903 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.191904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.192669 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.192671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.193185 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.193186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.193256 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.193260 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.193319 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.193360 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260925 14:58:10.193693 9867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.191118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57045 (local address 127.8.13.195:37793)
0925 14:58:10.191170 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.191172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.191179 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:10.191205 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.191206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.191214 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.191265 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:10.191400 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.191918 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.192822 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.193265 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.193328 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.193335 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.193560 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.193616 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:58:10.193728 9659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68b04e5c0a248bfa206356b1a9d5e82" candidate_uuid: "83c405c145624b69991819708616e89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf9a94715b14866be80f85e02efd845" is_pre_election: true
I20260925 14:58:10.193871 9659 raft_consensus.cc:2454] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c405c145624b69991819708616e89b in term 0.
I20260925 14:58:10.193882 9897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.191007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:37793 (local address 127.8.13.193:57045)
0925 14:58:10.191091 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.191100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.191155 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.191269 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.191269 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.191271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:10.191339 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.191341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.191988 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.191988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.192750 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.192752 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.193404 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.193405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.193465 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.193468 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.193768 (+ 300us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.193811 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"thread_start_us":38,"threads_started":1}
I20260925 14:58:10.194082 9453 leader_election.cc:302] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, 83c405c145624b69991819708616e89b; no voters:
I20260925 14:58:10.194093 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68b04e5c0a248bfa206356b1a9d5e82" candidate_uuid: "83c405c145624b69991819708616e89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee" is_pre_election: true
I20260925 14:58:10.194164 9799 raft_consensus.cc:2454] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83c405c145624b69991819708616e89b in term 0.
I20260925 14:58:10.194212 9872 raft_consensus.cc:2781] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:10.194253 9872 raft_consensus.cc:482] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:10.194271 9872 raft_consensus.cc:3037] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.194919 9872 raft_consensus.cc:504] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.195029 9872 leader_election.cc:288] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [CANDIDATE]: Term 1 election: Requested vote from peers 6cf9a94715b14866be80f85e02efd845 (127.8.13.194:41815), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:10.195196 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68b04e5c0a248bfa206356b1a9d5e82" candidate_uuid: "83c405c145624b69991819708616e89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee"
I20260925 14:58:10.195207 9659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68b04e5c0a248bfa206356b1a9d5e82" candidate_uuid: "83c405c145624b69991819708616e89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cf9a94715b14866be80f85e02efd845"
I20260925 14:58:10.195263 9659 raft_consensus.cc:3037] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.195259 9799 raft_consensus.cc:3037] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.195888 9799 raft_consensus.cc:2454] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c405c145624b69991819708616e89b in term 1.
I20260925 14:58:10.195895 9659 raft_consensus.cc:2454] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83c405c145624b69991819708616e89b in term 1.
I20260925 14:58:10.196053 9453 leader_election.cc:302] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, 83c405c145624b69991819708616e89b; no voters:
I20260925 14:58:10.196126 9872 raft_consensus.cc:2781] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:10.196235 9872 raft_consensus.cc:686] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 1 LEADER]: Becoming Leader. State: Replica: 83c405c145624b69991819708616e89b, State: Running, Role: LEADER
I20260925 14:58:10.196297 9872 consensus_queue.cc:237] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.196904 9362 catalog_manager.cc:5495] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b reported cstate change: term changed from 0 to 1, leader changed from <none> to 83c405c145624b69991819708616e89b (127.8.13.193). New cstate: current_term: 1 leader_uuid: "83c405c145624b69991819708616e89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } health_report { overall_health: UNKNOWN } } }
W20260925 14:58:10.260248 9566 tablet.cc:2369] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:58:10 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:10.290897 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.284284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43557 (local address 127.0.0.1:39434)
0925 14:58:10.284445 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.284455 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.284530 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.284667 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.284667 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.284952 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:10.285001 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.285003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.285611 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.285612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.286157 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.286158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.286482 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.286483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.286538 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.286969 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:10.286973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:10.288292 (+ 1319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:10.289857 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:10.289864 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:10.289866 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:10.290081 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:10.290297 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:10.290298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:10.290298 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:10.290348 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:10.290608 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:10.290614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:10.290659 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.290711 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.290802 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260925 14:58:10.290943 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.284361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39434 (local address 127.8.13.254:43557)
0925 14:58:10.284435 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.284437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.284541 (+ 104us) server_negotiation.cc:407] Connection header received
0925 14:58:10.284594 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.284595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.284604 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.284652 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:10.285073 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.285548 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.286238 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.286420 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.286472 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.288402 (+ 1930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:10.288403 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:10.288404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:10.288409 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:10.289783 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.290141 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.290142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.290142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.290209 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.290392 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.290393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.290393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.290551 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:10.290608 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.290737 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.290850 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":49}
I20260925 14:58:10.292773 9362 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:39434:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:58:10.292986 9362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:10.294528 9639 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5527be98542fb8474f5e84c5ca8d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c6003dc9aa6424a9620050af67b4b84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:10.294525 9499 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5527be98542fb8474f5e84c5ca8d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c6003dc9aa6424a9620050af67b4b84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:10.294574 9779 tablet_service.cc:1467] Processing CreateTablet for tablet 9df5527be98542fb8474f5e84c5ca8d1 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=3c6003dc9aa6424a9620050af67b4b84]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:10.294696 9499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5527be98542fb8474f5e84c5ca8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.294696 9639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5527be98542fb8474f5e84c5ca8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.294713 9779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9df5527be98542fb8474f5e84c5ca8d1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:10.296085 9869 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: Bootstrap starting.
I20260925 14:58:10.296205 9868 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: Bootstrap starting.
I20260925 14:58:10.296205 9873 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: Bootstrap starting.
I20260925 14:58:10.296476 9869 tablet_bootstrap.cc:654] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.296566 9873 tablet_bootstrap.cc:654] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.296566 9868 tablet_bootstrap.cc:654] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:10.297196 9868 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: No bootstrap required, opened a new log
I20260925 14:58:10.297243 9868 ts_tablet_manager.cc:1397] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:10.297257 9873 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: No bootstrap required, opened a new log
I20260925 14:58:10.297310 9873 ts_tablet_manager.cc:1397] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:10.297392 9868 raft_consensus.cc:348] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297425 9873 raft_consensus.cc:348] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297434 9869 tablet_bootstrap.cc:492] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: No bootstrap required, opened a new log
I20260925 14:58:10.297465 9868 raft_consensus.cc:374] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.297485 9868 raft_consensus.cc:729] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c405c145624b69991819708616e89b, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.297480 9873 raft_consensus.cc:374] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.297498 9869 ts_tablet_manager.cc:1397] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:10.297513 9873 raft_consensus.cc:729] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cf9a94715b14866be80f85e02efd845, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.297535 9868 consensus_queue.cc:260] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297560 9873 consensus_queue.cc:260] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297631 9868 ts_tablet_manager.cc:1428] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:10.297647 9873 ts_tablet_manager.cc:1428] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:10.297650 9869 raft_consensus.cc:348] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297706 9869 raft_consensus.cc:374] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:10.297667 9878 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.297724 9869 raft_consensus.cc:729] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9c807017ce24a4e8617cf28c054bbee, State: Initialized, Role: FOLLOWER
I20260925 14:58:10.297734 9878 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.297758 9869 consensus_queue.cc:260] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.297744 9872 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.297797 9872 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.297833 9869 ts_tablet_manager.cc:1428] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:10.298123 9875 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.298228 9875 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.644552 9878 raft_consensus.cc:482] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:10.644666 9878 raft_consensus.cc:504] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.644824 9878 leader_election.cc:288] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 83c405c145624b69991819708616e89b (127.8.13.193:33997), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:10.645076 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5527be98542fb8474f5e84c5ca8d1" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c405c145624b69991819708616e89b" is_pre_election: true
I20260925 14:58:10.645061 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5527be98542fb8474f5e84c5ca8d1" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee" is_pre_election: true
I20260925 14:58:10.645164 9519 raft_consensus.cc:2454] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 0.
I20260925 14:58:10.645167 9799 raft_consensus.cc:2454] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 0.
I20260925 14:58:10.645311 9592 leader_election.cc:302] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, 83c405c145624b69991819708616e89b; no voters:
I20260925 14:58:10.645437 9878 raft_consensus.cc:2781] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:10.645486 9878 raft_consensus.cc:482] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:10.645511 9878 raft_consensus.cc:3037] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.646212 9878 raft_consensus.cc:504] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.646332 9878 leader_election.cc:288] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 election: Requested vote from peers 83c405c145624b69991819708616e89b (127.8.13.193:33997), d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793)
I20260925 14:58:10.646509 9519 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5527be98542fb8474f5e84c5ca8d1" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c405c145624b69991819708616e89b"
I20260925 14:58:10.646523 9799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9df5527be98542fb8474f5e84c5ca8d1" candidate_uuid: "6cf9a94715b14866be80f85e02efd845" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9c807017ce24a4e8617cf28c054bbee"
I20260925 14:58:10.646600 9519 raft_consensus.cc:3037] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.646610 9799 raft_consensus.cc:3037] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:10.647320 9519 raft_consensus.cc:2454] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 1.
I20260925 14:58:10.647320 9799 raft_consensus.cc:2454] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6cf9a94715b14866be80f85e02efd845 in term 1.
I20260925 14:58:10.647495 9593 leader_election.cc:302] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cf9a94715b14866be80f85e02efd845, d9c807017ce24a4e8617cf28c054bbee; no voters:
I20260925 14:58:10.647609 9878 raft_consensus.cc:2781] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:10.647730 9878 raft_consensus.cc:686] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [term 1 LEADER]: Becoming Leader. State: Replica: 6cf9a94715b14866be80f85e02efd845, State: Running, Role: LEADER
I20260925 14:58:10.647794 9878 consensus_queue.cc:237] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } }
I20260925 14:58:10.648065 9909 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: TxnStatusTablet state changed. Reason: New leader 6cf9a94715b14866be80f85e02efd845. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.648146 9909 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:10.648346 9911 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:58:10.648391 9362 catalog_manager.cc:5495] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6cf9a94715b14866be80f85e02efd845 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:10.648470 9911 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:58:10.698691 9872 consensus_queue.cc:1035] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.700546 9872 consensus_queue.cc:1035] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.739912 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.737431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55140 (local address 127.8.13.194:41815)
0925 14:58:10.737612 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.737614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.737620 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:58:10.737653 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.737654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.737663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.737715 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:10.737877 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.738413 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.739179 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.739398 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.739451 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.739642 (+ 191us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.739681 (+ 39us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.739721 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.739797 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.739822 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260925 14:58:10.739908 9905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.737355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41815 (local address 127.0.0.1:55140)
0925 14:58:10.737438 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.737451 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.737507 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.737758 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.737759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.737760 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:10.737811 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.737813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.738491 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.738492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.739090 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.739092 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.739460 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.739461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.739509 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.739523 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.739733 (+ 210us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.739741 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.739781 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.739831 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:58:10.741303 9799 raft_consensus.cc:1261] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Refusing update from remote peer 6cf9a94715b14866be80f85e02efd845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:10.741317 9519 raft_consensus.cc:1261] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Refusing update from remote peer 6cf9a94715b14866be80f85e02efd845: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:10.741534 9909 consensus_queue.cc:1035] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.741665 9878 consensus_queue.cc:1035] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:10.742141 9872 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: TxnStatusTablet state changed. Reason: New leader 6cf9a94715b14866be80f85e02efd845. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.742161 9875 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: TxnStatusTablet state changed. Reason: New leader 6cf9a94715b14866be80f85e02efd845. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.742228 9872 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.742239 9875 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.742883 9878 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.742970 9878 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:10.742970 9915 mvcc.cc:205] Tried to move back new op lower bound from 7333266598874677248 to 7333266598493728768. Current Snapshot: MvccSnapshot[applied={T|T < 7333266598874677248}]
I20260925 14:58:10.743067 9872 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.743136 9872 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 83c405c145624b69991819708616e89b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.743217 9918 mvcc.cc:205] Tried to move back new op lower bound from 7333266598874677248 to 7333266598493728768. Current Snapshot: MvccSnapshot[applied={T|T < 7333266598874677248}]
I20260925 14:58:10.744148 9909 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.744227 9909 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P 6cf9a94715b14866be80f85e02efd845: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:10.744644 9875 tablet_replica.cc:439] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6cf9a94715b14866be80f85e02efd845" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "83c405c145624b69991819708616e89b" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33997 } } peers { permanent_uuid: "6cf9a94715b14866be80f85e02efd845" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41815 } } peers { permanent_uuid: "d9c807017ce24a4e8617cf28c054bbee" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37793 } } }
I20260925 14:58:10.744750 9875 tablet_replica.cc:442] T 9df5527be98542fb8474f5e84c5ca8d1 P d9c807017ce24a4e8617cf28c054bbee: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:10.748543 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.745491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.8.13.193:33997)
0925 14:58:10.745629 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.745631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.745664 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:10.745705 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.745706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.745728 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.745805 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:10.745970 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.746780 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.747569 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.747881 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.747952 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.748080 (+ 128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.748214 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.748284 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.748330 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.748354 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260925 14:58:10.753985 9932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.751423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41815 (local address 127.8.13.193:57427)
0925 14:58:10.751593 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.751603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.751662 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.751786 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.751786 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.751787 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:10.751831 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.751833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.752645 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.752646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.753224 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.753226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.753653 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.753654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.753697 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.753712 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.753850 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.753852 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.753879 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.753914 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260925 14:58:10.753985 9920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.751510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57427 (local address 127.8.13.194:41815)
0925 14:58:10.751582 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.751584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.751664 (+ 80us) server_negotiation.cc:407] Connection header received
0925 14:58:10.751707 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.751708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.751719 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.751774 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:10.751865 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.752580 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.753290 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.753587 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.753650 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.753775 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.753807 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.753848 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.753882 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.753904 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 14:58:10.764235 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.761386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:39459 (local address 127.8.13.193:33997)
0925 14:58:10.761427 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.761429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.761644 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:58:10.761720 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.761722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.761732 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.761776 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:10.761877 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.762337 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.763495 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.763816 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.763881 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.764038 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.764068 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.764100 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.764156 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.764174 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:10.764285 9935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.761342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33997 (local address 127.8.13.194:39459)
0925 14:58:10.761584 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:10.761596 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:10.761669 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:10.761779 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:10.761780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:10.761781 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:10.761830 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.761832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.762395 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.762396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.763424 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:10.763427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.763886 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.763886 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:10.763943 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.763960 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:10.764110 (+ 150us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:10.764111 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:10.764152 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:58:10.764198 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":85,"threads_started":1}
I20260925 14:58:10.773387 9362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:10.773504 9362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:10.774138 9362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf1144fb821947758f293561b9d23716: Sending DeleteTablet for 3 replicas of tablet a68b04e5c0a248bfa206356b1a9d5e82
I20260925 14:58:10.774362 9499 tablet_service.cc:1514] Processing DeleteTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:43960
I20260925 14:58:10.774407 9779 tablet_service.cc:1514] Processing DeleteTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50148
I20260925 14:58:10.774591 9639 tablet_service.cc:1514] Processing DeleteTablet for tablet a68b04e5c0a248bfa206356b1a9d5e82 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:10 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:55110
I20260925 14:58:10.774766 9940 tablet_replica.cc:330] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: stopping tablet replica
I20260925 14:58:10.774760 9938 tablet_replica.cc:330] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: stopping tablet replica
I20260925 14:58:10.774827 9940 raft_consensus.cc:2229] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:10.774869 9940 raft_consensus.cc:2258] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.774935 9938 raft_consensus.cc:2229] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:10.774971 9939 tablet_replica.cc:330] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: stopping tablet replica
I20260925 14:58:10.775070 9939 raft_consensus.cc:2229] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:10.775116 9939 raft_consensus.cc:2258] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.775091 9938 raft_consensus.cc:2258] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:10.775378 9940 ts_tablet_manager.cc:1905] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.775421 9938 ts_tablet_manager.cc:1905] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.776904 9939 ts_tablet_manager.cc:1905] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:10.777163 9938 ts_tablet_manager.cc:1918] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:10.777236 9938 log.cc:1196] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/a68b04e5c0a248bfa206356b1a9d5e82
I20260925 14:58:10.777438 9938 ts_tablet_manager.cc:1939] T a68b04e5c0a248bfa206356b1a9d5e82 P 83c405c145624b69991819708616e89b: Deleting consensus metadata
I20260925 14:58:10.777884 9348 catalog_manager.cc:4841] TS 83c405c145624b69991819708616e89b (127.8.13.193:33997): tablet a68b04e5c0a248bfa206356b1a9d5e82 (table test-table [id=b82fbadb1c524b41903707bf9f43ff2f]) successfully deleted
I20260925 14:58:10.778286 9939 ts_tablet_manager.cc:1918] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:10.778337 9939 log.cc:1196] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/a68b04e5c0a248bfa206356b1a9d5e82
I20260925 14:58:10.778513 9939 ts_tablet_manager.cc:1939] T a68b04e5c0a248bfa206356b1a9d5e82 P d9c807017ce24a4e8617cf28c054bbee: Deleting consensus metadata
I20260925 14:58:10.778663 9940 ts_tablet_manager.cc:1918] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:10.778713 9940 log.cc:1196] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/a68b04e5c0a248bfa206356b1a9d5e82
I20260925 14:58:10.778882 9940 ts_tablet_manager.cc:1939] T a68b04e5c0a248bfa206356b1a9d5e82 P 6cf9a94715b14866be80f85e02efd845: Deleting consensus metadata
I20260925 14:58:10.779222 9349 catalog_manager.cc:4841] TS 6cf9a94715b14866be80f85e02efd845 (127.8.13.194:41815): tablet a68b04e5c0a248bfa206356b1a9d5e82 (table test-table [id=b82fbadb1c524b41903707bf9f43ff2f]) successfully deleted
I20260925 14:58:10.779307 9349 catalog_manager.cc:4841] TS d9c807017ce24a4e8617cf28c054bbee (127.8.13.195:37793): tablet a68b04e5c0a248bfa206356b1a9d5e82 (table test-table [id=b82fbadb1c524b41903707bf9f43ff2f]) successfully deleted
Sep 25 14:58:10 dist-test-slave-15hz krb5kdc[9317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348289, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:10.786862 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.775741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.8.13.193:33997)
0925 14:58:10.775782 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.775785 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.777971 (+ 2186us) server_negotiation.cc:407] Connection header received
0925 14:58:10.778109 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.778110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.778124 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.778194 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:10.778904 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.779625 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.780648 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.780826 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.780882 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.784085 (+ 3203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:10.784093 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:10.784094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:10.784102 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:10.785697 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.786050 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.786054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.786054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.786121 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.786302 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.786302 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.786303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.786456 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:10.786649 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.786685 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.786760 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 14:58:10.787102 9559 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43986
I20260925 14:58:10.793879 9411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:10.788079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.8.13.254:43557)
0925 14:58:10.788129 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:10.788131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:10.788139 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:10.788193 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:10.788194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:10.788204 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:10.788248 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:10.788724 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.789227 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.790006 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:10.790212 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:10.790268 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:10.791443 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:10.791445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:10.791445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:10.791453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:10.792809 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.793131 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.793132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.793133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.793193 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:10.793361 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:10.793362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:10.793362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:10.793480 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:10.793567 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:10.793703 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:58:10.793785 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:58:10.794203 9362 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39448
I20260925 14:58:10.794864 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9434
I20260925 14:58:10.799885 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9574
I20260925 14:58:10.804783 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9714
I20260925 14:58:10.808840 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9330
2026-09-25T14:58:10Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2712 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:10 dist-test-slave-15hz krb5kdc[9947](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:10 dist-test-slave-15hz krb5kdc[9947](info): set up 2 sockets
Sep 25 14:58:10 dist-test-slave-15hz krb5kdc[9947](info): commencing operation
krb5kdc: starting...
W20260925 14:58:11.766979 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:11 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:11.881788 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33389
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:33389
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:11.961484 9960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:11.961715 9960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:11.961746 9960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:11.962924 9960 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:11.962967 9960 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:58:11.962983 9960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:11.963001 9960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:11.963022 9960 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:11.963038 9960 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:11.964555 9960 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:33389
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33389
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:11.964821 9960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:11.965040 9960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:11.969609 9968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:11.969588 9966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:11.969789 9960 server_base.cc:1034] running on GCE node
W20260925 14:58:11.969668 9965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:11.970119 9960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:11.970376 9960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:11.971520 9960 hybrid_clock.cc:648] HybridClock initialized: now 1790348291971498 us; error 35 us; skew 500 ppm
Sep 25 14:58:11 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:11.973904 9960 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:11.974412 9960 webserver.cc:460] Webserver started at http://127.8.13.254:37777/ using document root <none> and password file <none>
I20260925 14:58:11.974596 9960 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:11.974634 9960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:11.974726 9960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:11.975824 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "32c428adf3cc433eb6fa672c19f5b4eb"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-15hz"
server_key: "5e3138915b919c85e8fd1b6aa0429e03"
server_key_iv: "7d3c0c47d8ffcc46df8e6922d6844453"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.976137 9960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "32c428adf3cc433eb6fa672c19f5b4eb"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-15hz"
server_key: "5e3138915b919c85e8fd1b6aa0429e03"
server_key_iv: "7d3c0c47d8ffcc46df8e6922d6844453"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.977375 9960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:11.978099 9975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:11.978273 9960 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:11.978355 9960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "32c428adf3cc433eb6fa672c19f5b4eb"
format_stamp: "Formatted at 2026-09-25 14:58:11 on dist-test-slave-15hz"
server_key: "5e3138915b919c85e8fd1b6aa0429e03"
server_key_iv: "7d3c0c47d8ffcc46df8e6922d6844453"
server_key_version: "encryptionkey@0"
I20260925 14:58:11.978412 9960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:11.993852 9960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:11.996052 9960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:11.996212 9960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:12.000506 9960 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:33389
I20260925 14:58:12.000571 10037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:33389 every 8 connection(s)
I20260925 14:58:12.000871 9960 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:12.001547 10038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.003830 10038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Bootstrap starting.
I20260925 14:58:12.004379 10038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.004635 10038 log.cc:824] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:12.005349 10038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: No bootstrap required, opened a new log
I20260925 14:58:12.006394 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 9960
I20260925 14:58:12.006465 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:12.006608 8247 external_mini_cluster.cc:1428] Setting key 741b12bb71bbb6afc2d731408a68b429
I20260925 14:58:12.007786 10038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } }
I20260925 14:58:12.007978 10038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.008021 10038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32c428adf3cc433eb6fa672c19f5b4eb, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.008152 10038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } }
I20260925 14:58:12.008227 10038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:12.008251 10038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:12.008297 10038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.008946 10038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } }
I20260925 14:58:12.009063 10038 leader_election.cc:302] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32c428adf3cc433eb6fa672c19f5b4eb; no voters:
I20260925 14:58:12.009222 10038 leader_election.cc:288] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:12.009275 10043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:12.009425 10043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [term 1 LEADER]: Becoming Leader. State: Replica: 32c428adf3cc433eb6fa672c19f5b4eb, State: Running, Role: LEADER
I20260925 14:58:12.009469 10038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:12.009538 10043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } }
I20260925 14:58:12.009889 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32c428adf3cc433eb6fa672c19f5b4eb. Latest consensus state: current_term: 1 leader_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } } }
I20260925 14:58:12.009963 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:12.010100 10045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32c428adf3cc433eb6fa672c19f5b4eb" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33389 } } }
I20260925 14:58:12.010161 10045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:12.011130 10059 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:12.011184 10059 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:12.011230 10051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:12.011360 10051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:12.012905 10051 catalog_manager.cc:1347] Generated new cluster ID: 31ee6e7b1b30415f8ac8e179dc674686
I20260925 14:58:12.012961 10051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.017446 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.007252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34104 (local address 127.8.13.254:33389)
0925 14:58:12.007463 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.007475 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.007510 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:58:12.007754 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.007763 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.007825 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.007978 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.008719 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.010068 (+ 1349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.011121 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.011692 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.011791 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.014282 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.014286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.014287 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.014299 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.015911 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.016238 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.016241 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.016242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.016334 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.016544 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.016546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.016546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.016816 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.016963 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.017095 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.017242 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":90,"threads_started":1}
I20260925 14:58:12.022475 10051 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:12.023206 10051 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:12.028380 10051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Generated new TSK 0
I20260925 14:58:12.028584 10051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:12.060668 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33389
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:12.148080 10064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:12.148295 10064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:12.148330 10064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:12.149585 10064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:12.149641 10064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:12.149672 10064 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:12.149688 10064 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:12.149711 10064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:12.151278 10064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:12.151571 10064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:12.151804 10064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:12.156668 10072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:12.156731 10070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.156744 10064 server_base.cc:1034] running on GCE node
W20260925 14:58:12.156898 10069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.157124 10064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:12.157357 10064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:12.158505 10064 hybrid_clock.cc:648] HybridClock initialized: now 1790348292158490 us; error 24 us; skew 500 ppm
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:12.160926 10064 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:12.161461 10064 webserver.cc:460] Webserver started at http://127.8.13.193:35345/ using document root <none> and password file <none>
I20260925 14:58:12.161680 10064 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:12.161728 10064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:12.161854 10064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:12.162905 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "035a2c86749343efbe2083f516cc7c72"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "23a35ebac14ed65c1009aceb35c92802"
server_key_iv: "e63db5f00f07cfcb65419117f0245eca"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.163240 10064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "035a2c86749343efbe2083f516cc7c72"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "23a35ebac14ed65c1009aceb35c92802"
server_key_iv: "e63db5f00f07cfcb65419117f0245eca"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.164498 10064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:12.165151 10079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.165301 10064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:12.165380 10064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "035a2c86749343efbe2083f516cc7c72"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "23a35ebac14ed65c1009aceb35c92802"
server_key_iv: "e63db5f00f07cfcb65419117f0245eca"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.165438 10064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:12.186053 10064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:12.188313 10064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:12.188436 10064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:12.188902 10064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:12.188941 10064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.188966 10064 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:12.188977 10064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.195571 10064 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:44077
I20260925 14:58:12.195644 10194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:44077 every 8 connection(s)
I20260925 14:58:12.195945 10064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:12.196381 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10064
I20260925 14:58:12.196476 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:12.196640 8247 external_mini_cluster.cc:1428] Setting key 09897490eb64fc763a2386c11fe30228
I20260925 14:58:12.198571 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.189501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:48617 (local address 127.8.13.254:33389)
0925 14:58:12.189569 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.189572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.189772 (+ 200us) server_negotiation.cc:407] Connection header received
0925 14:58:12.190324 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.190325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.190341 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.190398 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.191261 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.192084 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.193205 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.193415 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.193472 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.195786 (+ 2314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.195787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.195788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.195797 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.197313 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.197672 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.197673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.197673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.197733 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.197954 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.197955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.197955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.198069 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.198123 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.198395 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.198481 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:12.198745 10088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.189467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.193:48617)
0925 14:58:12.189635 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.189679 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.190219 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.190478 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.190483 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.190843 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.191148 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.191156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.192195 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.192196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.193103 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.193106 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.193508 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.193509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.193646 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.194185 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.194194 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.195694 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.197380 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.197386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.197388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.197599 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.197787 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.197788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.197788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.197896 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.198125 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.198131 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.198270 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.198358 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.198520 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":98,"threads_started":1}
I20260925 14:58:12.199219 10195 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33389
I20260925 14:58:12.199298 10195 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:12.199479 10195 heartbeater.cc:511] Master 127.8.13.254:33389 requested a full tablet report, sending...
I20260925 14:58:12.200024 9991 ts_manager.cc:194] Registered new tserver with Master: 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077)
I20260925 14:58:12.200635 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:48617
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:12.206570 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.201084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:40859 (local address 127.8.13.254:33389)
0925 14:58:12.201119 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.201121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.201129 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:12.201213 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.201214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.201224 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.201279 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.201644 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.202096 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.203056 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.203246 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.203286 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.204297 (+ 1011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.204298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.204298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.204304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.205520 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.205856 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.205857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.205857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.205915 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.206176 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.206176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.206177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.206274 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.206319 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.206452 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.206508 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:58:12.206611 10202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.200920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.193:40859)
0925 14:58:12.201057 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.201068 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.201122 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.201295 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.201296 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.201485 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.201566 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.201568 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.202189 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.202190 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.202984 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.202986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.203308 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.203309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.203385 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.203830 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.203836 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.204215 (+ 379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.205608 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.205609 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.205609 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.205784 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.206023 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.206024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.206024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.206095 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.206324 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.206325 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.206376 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.206435 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.206535 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":87,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:12.220633 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33389
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:12.301143 10204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:12.301350 10204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:12.301374 10204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:12.302534 10204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:12.302580 10204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:12.302600 10204 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:12.302611 10204 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:12.302627 10204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:12.304090 10204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10204
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:12.304301 10204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:12.304519 10204 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:12.309270 10210 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:12.309288 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.309355 10204 server_base.cc:1034] running on GCE node
W20260925 14:58:12.309300 10212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.309715 10204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:12.309950 10204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:12.311125 10204 hybrid_clock.cc:648] HybridClock initialized: now 1790348292311125 us; error 46 us; skew 500 ppm
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:12.313539 10204 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:12.313967 10204 webserver.cc:460] Webserver started at http://127.8.13.194:39531/ using document root <none> and password file <none>
I20260925 14:58:12.314198 10204 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:12.314247 10204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:12.314373 10204 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:12.315430 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "849223981c464676b5a7240b7660c629"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "06fe8ee7de557d8889172ec9ea9f7a74"
server_key_iv: "f72313655a76bfc6fe28cd356d15b5b8"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.315819 10204 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "849223981c464676b5a7240b7660c629"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "06fe8ee7de557d8889172ec9ea9f7a74"
server_key_iv: "f72313655a76bfc6fe28cd356d15b5b8"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.317101 10204 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:12.317819 10219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.317976 10204 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:12.318089 10204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "849223981c464676b5a7240b7660c629"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "06fe8ee7de557d8889172ec9ea9f7a74"
server_key_iv: "f72313655a76bfc6fe28cd356d15b5b8"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.318154 10204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:12.335363 10204 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:12.337500 10204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:12.337651 10204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:12.338299 10204 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:12.338347 10204 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.338369 10204 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:12.338382 10204 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.344589 10204 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:44315
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.344668 10334 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:44315 every 8 connection(s)
I20260925 14:58:12.344967 10204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:12.345372 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10204
I20260925 14:58:12.345465 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:12.345659 8247 external_mini_cluster.cc:1428] Setting key 2cd4a4cdf47f57a2a33d04e3c0b5505e
I20260925 14:58:12.347884 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.338830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:53419 (local address 127.8.13.254:33389)
0925 14:58:12.338898 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.338901 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.339076 (+ 175us) server_negotiation.cc:407] Connection header received
0925 14:58:12.339642 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.339644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.339657 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.339706 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.340534 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.341110 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.342611 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.342843 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.342902 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.345169 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.345171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.345172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.345181 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.346614 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.346973 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.346975 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.346975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.347039 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.347253 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.347253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.347254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.347375 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.347434 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.347710 (+ 276us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.347793 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:12.348054 10228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.338780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.194:53419)
0925 14:58:12.338962 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.339002 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.339508 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.339781 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.339786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.340128 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.340419 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.340426 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.341199 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.341200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.342504 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.342507 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.342921 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.342922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.343051 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.343630 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.343640 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.345058 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.346689 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.346695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.346696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.346901 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.347103 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.347103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.347104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.347198 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.347425 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.347428 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.347588 (+ 160us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.347679 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.347828 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":92,"threads_started":1}
I20260925 14:58:12.348541 10335 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33389
I20260925 14:58:12.348623 10335 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:12.348806 10335 heartbeater.cc:511] Master 127.8.13.254:33389 requested a full tablet report, sending...
I20260925 14:58:12.349201 9992 ts_manager.cc:194] Registered new tserver with Master: 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:12.349669 9992 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:53419
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:12.355417 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.349571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:43173 (local address 127.8.13.254:33389)
0925 14:58:12.349609 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.349611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.349713 (+ 102us) server_negotiation.cc:407] Connection header received
0925 14:58:12.349817 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.349818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.349829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.349856 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.350188 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.350672 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.351681 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.351862 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.351914 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.352921 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.352922 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.352923 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.352929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.354298 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.354670 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.354670 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.354671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.354720 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.354912 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.354912 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.354913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.355090 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.355150 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.355295 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.355353 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:58:12.355427 10342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.349512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.194:43173)
0925 14:58:12.349651 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.349660 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.349712 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.349909 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.349910 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.350064 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.350119 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.350121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.350778 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.350779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.351590 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.351593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.351936 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.351937 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.352017 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.352385 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.352390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.352800 (+ 410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.354380 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.354381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.354382 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.354594 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.354783 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.354783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.354784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.354847 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.355164 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.355165 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.355216 (+ 51us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.355271 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.355353 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:12.374528 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33389
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:37887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:12.453611 10344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:12.453828 10344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:12.453855 10344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:12.455054 10344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:12.455099 10344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:12.455120 10344 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260925 14:58:12.455130 10344 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:12.455147 10344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:12.456707 10344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33389
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:12.456944 10344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:12.457170 10344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:12.461933 10350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:12.461933 10349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.462127 10344 server_base.cc:1034] running on GCE node
W20260925 14:58:12.461973 10352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:12.462391 10344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:12.462644 10344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:12.463794 10344 hybrid_clock.cc:648] HybridClock initialized: now 1790348292463780 us; error 31 us; skew 500 ppm
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:12.466215 10344 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:12.466737 10344 webserver.cc:460] Webserver started at http://127.8.13.195:32991/ using document root <none> and password file <none>
I20260925 14:58:12.466933 10344 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:12.466975 10344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:12.467097 10344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:12.468187 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "c193d040a7b746708db40851cbfa42f7"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "259fbd4113222f1bfd1552fb31107719"
server_key_iv: "17b6ebdebb710d6c14bd108d356b9878"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.468523 10344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "c193d040a7b746708db40851cbfa42f7"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "259fbd4113222f1bfd1552fb31107719"
server_key_iv: "17b6ebdebb710d6c14bd108d356b9878"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.469751 10344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:12.470407 10359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.470573 10344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:12.470642 10344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "c193d040a7b746708db40851cbfa42f7"
format_stamp: "Formatted at 2026-09-25 14:58:12 on dist-test-slave-15hz"
server_key: "259fbd4113222f1bfd1552fb31107719"
server_key_iv: "17b6ebdebb710d6c14bd108d356b9878"
server_key_version: "encryptionkey@0"
I20260925 14:58:12.470691 10344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:12.484094 10344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:12.486222 10344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:12.486336 10344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:12.486830 10344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:12.486874 10344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.486905 10344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:12.486923 10344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.493708 10344 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:34685
I20260925 14:58:12.493745 10474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:34685 every 8 connection(s)
I20260925 14:58:12.494076 10344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:12.496459 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.487271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:58089 (local address 127.8.13.254:33389)
0925 14:58:12.487339 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.487341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.487522 (+ 181us) server_negotiation.cc:407] Connection header received
0925 14:58:12.488077 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.488078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.488092 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.488167 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.488935 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.489499 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.490511 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.490711 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.490770 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.493504 (+ 2734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.493505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.493505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.493514 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.495014 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.495467 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.495469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.495469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.495574 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.495795 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.495796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.495796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.495921 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.495974 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.496294 (+ 320us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.496383 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:12.496533 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.487240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.195:58089)
0925 14:58:12.487394 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.487435 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.487964 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.488241 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.488245 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.488605 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.488807 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.488812 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.489587 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.489588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.490419 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.490421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.490789 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.490790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.490930 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.491488 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.491499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.493376 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.495110 (+ 1734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.495115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.495117 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.495367 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.495630 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.495631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.495631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.495737 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.495976 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.495979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.496119 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.496218 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.496355 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260925 14:58:12.497071 10475 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33389
I20260925 14:58:12.497170 10475 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:12.497368 10475 heartbeater.cc:511] Master 127.8.13.254:33389 requested a full tablet report, sending...
I20260925 14:58:12.497766 9991 ts_manager.cc:194] Registered new tserver with Master: c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685)
I20260925 14:58:12.498189 9991 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:58089
I20260925 14:58:12.499184 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10344
I20260925 14:58:12.499271 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:12.499459 8247 external_mini_cluster.cc:1428] Setting key 0fb5976b39080531d73f78d11b3a5d33
I20260925 14:58:12.500375 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:12.504360 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.498115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.8.13.195:47029)
0925 14:58:12.498254 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.498264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.498325 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.498619 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.498620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.498810 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.498887 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.498889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.499739 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.499740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.500462 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.500465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.500788 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.500789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.500863 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.501418 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.501424 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.501795 (+ 371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.503362 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.503363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.503364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.503564 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.503749 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.503749 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.503749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.503800 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.504057 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.504058 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.504116 (+ 58us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.504182 (+ 66us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.504275 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":85,"threads_started":1}
I20260925 14:58:12.504385 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.498198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:47029 (local address 127.8.13.254:33389)
0925 14:58:12.498247 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.498249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.498311 (+ 62us) server_negotiation.cc:407] Connection header received
0925 14:58:12.498510 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.498510 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.498519 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.498565 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.498959 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.499643 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.500534 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.500718 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.500770 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.501880 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.501881 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.501881 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.501889 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.503286 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.503632 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.503633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.503633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.503693 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.503849 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.503850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.503850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.503972 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.504033 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.504220 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.504297 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348292, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.523471 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.516866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.8.13.254:33389)
0925 14:58:12.516919 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.516921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.516929 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:12.517008 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.517009 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.517019 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.517066 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.517728 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.518220 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.518940 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.519128 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.519188 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.521432 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.521433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.521434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.521441 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.522612 (+ 1171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.522908 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.522909 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.522909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.522958 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.523101 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.523101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.523101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.523202 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.523271 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.523349 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.523404 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:12.524945 9991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:12.526415 9991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:12.532953 10489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:44077 (local address 127.0.0.1:32942)
0925 14:58:12.529001 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.529045 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.529112 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.529560 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.529563 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.529565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.529639 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.529641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.530520 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.530521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.531599 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.531602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532658 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.532734 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.532749 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.532814 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.532871 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":57,"threads_started":1}
I20260925 14:58:12.533133 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.529145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34982 (local address 127.8.13.194:44315)
0925 14:58:12.529418 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.529422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.529431 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:12.529468 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.529470 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.529485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.529564 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.529819 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.530954 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532166 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532672 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532736 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.532788 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.532990 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.533030 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":89,"threads_started":1}
I20260925 14:58:12.533182 10492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44315 (local address 127.0.0.1:34982)
0925 14:58:12.529283 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.529292 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.529339 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.529706 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.529706 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.529709 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.529742 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.529743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.531060 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.531062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.532096 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.532098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532820 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.532873 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.532876 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.532917 (+ 41us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.532948 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":58,"threads_started":1}
I20260925 14:58:12.533308 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.0.0.1:52390)
0925 14:58:12.529613 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.529621 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.529665 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.529890 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.529891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.529893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.529944 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.529946 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.531145 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.531146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.532119 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.532122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.533035 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.533036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.533109 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.533113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.533156 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.533238 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":53,"threads_started":1}
I20260925 14:58:12.533548 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.529119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52390 (local address 127.8.13.195:34685)
0925 14:58:12.529280 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.529285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.529685 (+ 400us) server_negotiation.cc:407] Connection header received
0925 14:58:12.529728 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.529730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.529744 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.529811 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.530030 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.531064 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532206 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532890 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532952 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.533000 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.533410 (+ 410us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.533450 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260925 14:58:12.533937 10269 tablet_service.cc:1467] Processing CreateTablet for tablet 75a29540b579452681fc8579b6317dfc (DEFAULT_TABLE table=test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.534111 10409 tablet_service.cc:1467] Processing CreateTablet for tablet 75a29540b579452681fc8579b6317dfc (DEFAULT_TABLE table=test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.534276 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a29540b579452681fc8579b6317dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.534428 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a29540b579452681fc8579b6317dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.532979 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.528926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32942 (local address 127.8.13.193:44077)
0925 14:58:12.529356 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.529360 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.529370 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:12.529409 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.529411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.529427 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.529543 (+ 116us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.529717 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.530440 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.531693 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.532566 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.532637 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.532694 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.532840 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.532878 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":98,"threads_started":1}
I20260925 14:58:12.536906 10495 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Bootstrap starting.
I20260925 14:58:12.537232 10496 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Bootstrap starting.
I20260925 14:58:12.537761 10495 tablet_bootstrap.cc:654] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.537767 10496 tablet_bootstrap.cc:654] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.537657 10129 tablet_service.cc:1467] Processing CreateTablet for tablet 75a29540b579452681fc8579b6317dfc (DEFAULT_TABLE table=test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.538012 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75a29540b579452681fc8579b6317dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.538044 10496 log.cc:824] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:12.538066 10495 log.cc:824] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:12.538865 10496 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: No bootstrap required, opened a new log
I20260925 14:58:12.538923 10496 ts_tablet_manager.cc:1397] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:12.540331 10496 raft_consensus.cc:348] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.540458 10496 raft_consensus.cc:374] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.540490 10496 raft_consensus.cc:729] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849223981c464676b5a7240b7660c629, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.540603 10496 consensus_queue.cc:260] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.540861 10496 ts_tablet_manager.cc:1428] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:12.540889 10500 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Bootstrap starting.
I20260925 14:58:12.540983 10335 heartbeater.cc:503] Master 127.8.13.254:33389 was elected leader, sending a full tablet report...
I20260925 14:58:12.541404 10495 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: No bootstrap required, opened a new log
I20260925 14:58:12.541473 10495 ts_tablet_manager.cc:1397] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260925 14:58:12.541517 10500 tablet_bootstrap.cc:654] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.541862 10500 log.cc:824] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:12.543174 10495 raft_consensus.cc:348] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.543301 10495 raft_consensus.cc:374] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.543330 10495 raft_consensus.cc:729] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.543442 10495 consensus_queue.cc:260] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.543720 10495 ts_tablet_manager.cc:1428] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:12.543753 10475 heartbeater.cc:503] Master 127.8.13.254:33389 was elected leader, sending a full tablet report...
I20260925 14:58:12.544247 10500 tablet_bootstrap.cc:492] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: No bootstrap required, opened a new log
I20260925 14:58:12.544315 10500 ts_tablet_manager.cc:1397] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:12.545760 10500 raft_consensus.cc:348] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.545882 10500 raft_consensus.cc:374] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.545902 10500 raft_consensus.cc:729] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.545984 10500 consensus_queue.cc:260] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.546234 10195 heartbeater.cc:503] Master 127.8.13.254:33389 was elected leader, sending a full tablet report...
I20260925 14:58:12.546447 10500 ts_tablet_manager.cc:1428] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260925 14:58:12.596184 10336 tablet.cc:2369] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:12.696835 10196 tablet.cc:2369] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:12.728324 10502 raft_consensus.cc:482] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:12.728470 10502 raft_consensus.cc:504] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.728844 10502 leader_election.cc:288] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:12.732668 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.729175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:56803 (local address 127.8.13.193:44077)
0925 14:58:12.729213 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.729215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.729224 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:12.729277 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.729278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.729289 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.729339 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.729512 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.730258 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.731711 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.732254 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.732330 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.732339 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.732554 (+ 215us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.732595 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:12.732668 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.729128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:46557 (local address 127.8.13.194:44315)
0925 14:58:12.729172 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.729174 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.729338 (+ 164us) server_negotiation.cc:407] Connection header received
0925 14:58:12.729375 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.729377 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.729388 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.729436 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.729590 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.730269 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.731711 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.732304 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.732363 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.732370 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.732554 (+ 184us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.732594 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:12.732700 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.729094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:44077 (local address 127.8.13.195:56803)
0925 14:58:12.729160 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.729172 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.729216 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.729361 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.729362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.729363 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.729432 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.729434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.730342 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.730343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.731624 (+ 1281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.731628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.732361 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.732363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.732456 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.732461 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.732541 (+ 80us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.732594 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":17}
I20260925 14:58:12.732713 10504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.729097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44315 (local address 127.8.13.195:46557)
0925 14:58:12.729257 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.729266 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.729323 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.729450 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.729451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.729452 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.729526 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.729527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.730341 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.730342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.731624 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.731628 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.732394 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.732394 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.732463 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.732478 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.732541 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.732594 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":126,"thread_start_us":98,"threads_started":1}
I20260925 14:58:12.733081 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a29540b579452681fc8579b6317dfc" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72" is_pre_election: true
I20260925 14:58:12.733108 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a29540b579452681fc8579b6317dfc" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629" is_pre_election: true
I20260925 14:58:12.733230 10149 raft_consensus.cc:2454] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:12.733259 10289 raft_consensus.cc:2454] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:12.733487 10361 leader_election.cc:302] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849223981c464676b5a7240b7660c629, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:12.733671 10502 raft_consensus.cc:2781] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:12.733736 10502 raft_consensus.cc:482] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:12.733759 10502 raft_consensus.cc:3037] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.734463 10502 raft_consensus.cc:504] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.734589 10502 leader_election.cc:288] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Requested vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:12.734762 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a29540b579452681fc8579b6317dfc" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72"
I20260925 14:58:12.734848 10149 raft_consensus.cc:3037] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.735577 10149 raft_consensus.cc:2454] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:12.735695 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75a29540b579452681fc8579b6317dfc" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629"
I20260925 14:58:12.735777 10289 raft_consensus.cc:3037] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.736094 10362 leader_election.cc:302] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:12.736209 10502 raft_consensus.cc:2781] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:12.736394 10502 raft_consensus.cc:686] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 1 LEADER]: Becoming Leader. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Running, Role: LEADER
I20260925 14:58:12.736507 10502 consensus_queue.cc:237] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } }
I20260925 14:58:12.736660 10289 raft_consensus.cc:2454] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:12.737504 9991 catalog_manager.cc:5495] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c193d040a7b746708db40851cbfa42f7 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } health_report { overall_health: UNKNOWN } } }
W20260925 14:58:12.744874 10476 tablet.cc:2369] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 14:58:12 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348291, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:12.782814 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.776241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33389 (local address 127.0.0.1:34114)
0925 14:58:12.776422 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.776432 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.776500 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.776622 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.776622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.776909 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:12.776957 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.776960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.777565 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.777565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.778101 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.778103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.778441 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.778442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.778479 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.778993 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:12.778997 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:12.780370 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:12.781779 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.781786 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.781788 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.781978 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.782196 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:12.782196 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:12.782197 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:12.782246 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:12.782529 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:12.782534 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:12.782579 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.782630 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.782715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260925 14:58:12.782863 10042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.776306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.8.13.254:33389)
0925 14:58:12.776368 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.776370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.776493 (+ 123us) server_negotiation.cc:407] Connection header received
0925 14:58:12.776557 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.776558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.776567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.776604 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:12.777030 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.777506 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.778181 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.778378 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.778440 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.780464 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:12.780465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:12.780465 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:12.780471 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:12.781709 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.782068 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.782069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.782070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.782136 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:12.782341 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:12.782343 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:12.782343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:12.782465 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:12.782529 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.782696 (+ 167us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.782763 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:12.785015 9991 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:34114:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:58:12.785218 9991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:12.786792 10129 tablet_service.cc:1467] Processing CreateTablet for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fbf852b39d54c3086d45efb492d7d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:12.786803 10409 tablet_service.cc:1467] Processing CreateTablet for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fbf852b39d54c3086d45efb492d7d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:12.786831 10269 tablet_service.cc:1467] Processing CreateTablet for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=9fbf852b39d54c3086d45efb492d7d8f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:12.786962 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.786962 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.786969 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70c8c0197de1483b8e0d1a8ab7b8b38f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.788264 10496 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: Bootstrap starting.
I20260925 14:58:12.788456 10500 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: Bootstrap starting.
I20260925 14:58:12.788839 10500 tablet_bootstrap.cc:654] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.788951 10496 tablet_bootstrap.cc:654] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.789172 10495 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: Bootstrap starting.
I20260925 14:58:12.789561 10495 tablet_bootstrap.cc:654] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.789593 10496 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: No bootstrap required, opened a new log
I20260925 14:58:12.789630 10496 ts_tablet_manager.cc:1397] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:12.789790 10496 raft_consensus.cc:348] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.789870 10496 raft_consensus.cc:374] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.789889 10496 raft_consensus.cc:729] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849223981c464676b5a7240b7660c629, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.789932 10496 consensus_queue.cc:260] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.790014 10500 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: No bootstrap required, opened a new log
I20260925 14:58:12.790019 10496 ts_tablet_manager.cc:1428] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:12.790066 10500 ts_tablet_manager.cc:1397] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:12.790184 10499 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.790223 10500 raft_consensus.cc:348] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.790266 10499 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.790282 10500 raft_consensus.cc:374] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.790307 10500 raft_consensus.cc:729] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.790354 10500 consensus_queue.cc:260] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.790448 10500 ts_tablet_manager.cc:1428] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.790576 10503 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.790635 10495 tablet_bootstrap.cc:492] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: No bootstrap required, opened a new log
I20260925 14:58:12.790654 10503 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.790673 10495 ts_tablet_manager.cc:1397] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:12.790807 10495 raft_consensus.cc:348] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.790882 10495 raft_consensus.cc:374] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.790901 10495 raft_consensus.cc:729] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.790938 10495 consensus_queue.cc:260] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.791021 10495 ts_tablet_manager.cc:1428] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:12.791152 10502 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.791220 10502 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.817340 10502 raft_consensus.cc:482] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:12.817440 10502 raft_consensus.cc:504] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.817590 10502 leader_election.cc:288] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:12.817822 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c8c0197de1483b8e0d1a8ab7b8b38f" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72" is_pre_election: true
I20260925 14:58:12.817909 10149 raft_consensus.cc:2454] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:12.817811 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c8c0197de1483b8e0d1a8ab7b8b38f" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629" is_pre_election: true
I20260925 14:58:12.818018 10289 raft_consensus.cc:2454] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:12.818075 10362 leader_election.cc:302] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:12.818184 10502 raft_consensus.cc:2781] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:12.818235 10502 raft_consensus.cc:482] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:12.818254 10502 raft_consensus.cc:3037] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.818888 10502 raft_consensus.cc:504] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.819000 10502 leader_election.cc:288] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Requested vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:12.819231 10149 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c8c0197de1483b8e0d1a8ab7b8b38f" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72"
I20260925 14:58:12.819236 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70c8c0197de1483b8e0d1a8ab7b8b38f" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629"
I20260925 14:58:12.819303 10149 raft_consensus.cc:3037] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.819303 10289 raft_consensus.cc:3037] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.819937 10289 raft_consensus.cc:2454] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:12.819964 10149 raft_consensus.cc:2454] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:12.820137 10362 leader_election.cc:302] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:12.820236 10502 raft_consensus.cc:2781] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:12.820294 10502 raft_consensus.cc:686] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [term 1 LEADER]: Becoming Leader. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Running, Role: LEADER
I20260925 14:58:12.820348 10502 consensus_queue.cc:237] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.820653 10505 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: TxnStatusTablet state changed. Reason: New leader c193d040a7b746708db40851cbfa42f7. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.820722 10505 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:12.820902 10517 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:58:12.821000 10517 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:58:12.820945 9991 catalog_manager.cc:5495] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c193d040a7b746708db40851cbfa42f7 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:12.828444 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.825478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.8.13.195:34685)
0925 14:58:12.825525 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.825527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.825550 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:12.825636 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.825637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.825653 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.825702 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.825856 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.826340 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.827454 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.827732 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.827791 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.828060 (+ 269us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.828207 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.828290 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.828359 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.828383 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:58:12.828491 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.825401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.0.0.1:52392)
0925 14:58:12.825487 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.825499 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.825547 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.825735 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.825737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.825738 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:12.825801 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.825804 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.826421 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.826422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.827349 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.827354 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.827807 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.827808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.827881 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.827902 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.828279 (+ 377us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.828290 (+ 11us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.828348 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.828407 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:58:12.830394 10289 raft_consensus.cc:1261] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.830394 10149 raft_consensus.cc:1261] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.830777 10505 consensus_queue.cc:1035] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.830906 10502 consensus_queue.cc:1035] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.832154 10499 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: TxnStatusTablet state changed. Reason: New leader c193d040a7b746708db40851cbfa42f7. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.832267 10499 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.832232 10503 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: TxnStatusTablet state changed. Reason: New leader c193d040a7b746708db40851cbfa42f7. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.832314 10503 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.833209 10502 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.833300 10502 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:12.833427 10505 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.833495 10505 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P c193d040a7b746708db40851cbfa42f7: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:12.833884 10503 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.833968 10503 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 035a2c86749343efbe2083f516cc7c72: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.834002 10499 tablet_replica.cc:439] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } } }
I20260925 14:58:12.834074 10499 tablet_replica.cc:442] T 70c8c0197de1483b8e0d1a8ab7b8b38f P 849223981c464676b5a7240b7660c629: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:12.834249 10522 mvcc.cc:205] Tried to move back new op lower bound from 7333266607430758400 to 7333266607392571392. Current Snapshot: MvccSnapshot[applied={T|T < 7333266607430758400}]
I20260925 14:58:12.835315 10521 mvcc.cc:205] Tried to move back new op lower bound from 7333266607430758400 to 7333266607392571392. Current Snapshot: MvccSnapshot[applied={T|T < 7333266607430758400 or (T in {7333266607430758400})}]
I20260925 14:58:12.838018 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.835567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.8.13.195:34685)
0925 14:58:12.835602 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.835604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.835695 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:58:12.835718 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.835719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.835727 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.835771 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.835902 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.836357 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.837458 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.837646 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.837697 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.837841 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.837864 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.837892 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.837946 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.837961 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:12.843326 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.840731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.8.13.195:60171)
0925 14:58:12.840806 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.840817 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.840848 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.841058 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.841059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.841060 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:12.841109 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.841110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.841944 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.841945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.842528 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.842529 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.842955 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.842956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.842998 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.843037 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.843196 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.843197 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.843225 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.843258 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 14:58:12.843333 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.840896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:60171 (local address 127.8.13.195:34685)
0925 14:58:12.840945 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.840947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.840955 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:12.840974 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.840974 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.840984 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.841051 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.841171 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.841879 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.842602 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.842893 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.842959 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.843116 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.843146 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.843192 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.843230 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.843246 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:12.845729 10289 raft_consensus.cc:1261] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.845729 10149 raft_consensus.cc:1261] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.845957 10531 consensus_queue.cc:1035] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.846107 10502 consensus_queue.cc:1035] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.861090 9991 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:12.861200 9991 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:12.861927 9991 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Sending DeleteTablet for 3 replicas of tablet 75a29540b579452681fc8579b6317dfc
I20260925 14:58:12.862286 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a29540b579452681fc8579b6317dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:12 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:32942
I20260925 14:58:12.862313 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a29540b579452681fc8579b6317dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:12 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:52390
I20260925 14:58:12.862551 10269 tablet_service.cc:1514] Processing DeleteTablet for tablet 75a29540b579452681fc8579b6317dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:12 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:34982
I20260925 14:58:12.862613 10537 tablet_replica.cc:330] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: stopping tablet replica
I20260925 14:58:12.862721 10539 tablet_replica.cc:330] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: stopping tablet replica
I20260925 14:58:12.862730 10537 raft_consensus.cc:2229] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:12.862830 10537 raft_consensus.cc:2258] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:12.862838 10539 raft_consensus.cc:2229] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:12.862893 10539 raft_consensus.cc:2258] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:12.863072 10538 tablet_replica.cc:330] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: stopping tablet replica
I20260925 14:58:12.863173 10538 raft_consensus.cc:2229] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:12.863221 10538 raft_consensus.cc:2258] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:12.863273 9991 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:12.863441 9991 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:12.863446 10537 ts_tablet_manager.cc:1905] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:12.863811 10538 ts_tablet_manager.cc:1905] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:12.864270 10539 ts_tablet_manager.cc:1905] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:12.864897 10129 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 (DEFAULT_TABLE table=test-table [id=887e971f163c42bc86ccf6aec14a79e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.865017 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2dcc7d82694e46abab9151b5e48d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.865546 10538 ts_tablet_manager.cc:1918] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:12.865620 10538 log.cc:1196] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/75a29540b579452681fc8579b6317dfc
I20260925 14:58:12.865864 10538 ts_tablet_manager.cc:1939] T 75a29540b579452681fc8579b6317dfc P 035a2c86749343efbe2083f516cc7c72: Deleting consensus metadata
I20260925 14:58:12.866288 10500 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Bootstrap starting.
I20260925 14:58:12.866426 10539 ts_tablet_manager.cc:1918] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:12.866547 10539 log.cc:1196] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/75a29540b579452681fc8579b6317dfc
I20260925 14:58:12.866667 10500 tablet_bootstrap.cc:654] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.866765 10409 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 (DEFAULT_TABLE table=test-table [id=887e971f163c42bc86ccf6aec14a79e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.866875 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2dcc7d82694e46abab9151b5e48d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.867056 9978 catalog_manager.cc:4841] TS 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077): tablet 75a29540b579452681fc8579b6317dfc (table test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]) successfully deleted
I20260925 14:58:12.867065 10269 tablet_service.cc:1467] Processing CreateTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 (DEFAULT_TABLE table=test-table [id=887e971f163c42bc86ccf6aec14a79e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:12.867168 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b2dcc7d82694e46abab9151b5e48d95. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:12.867515 10500 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: No bootstrap required, opened a new log
I20260925 14:58:12.867580 10500 ts_tablet_manager.cc:1397] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:12.867707 10500 raft_consensus.cc:348] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.867769 10500 raft_consensus.cc:374] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.867787 10500 raft_consensus.cc:729] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.867826 10500 consensus_queue.cc:260] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.867910 10500 ts_tablet_manager.cc:1428] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.868283 10495 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Bootstrap starting.
I20260925 14:58:12.868355 10496 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Bootstrap starting.
I20260925 14:58:12.868692 10495 tablet_bootstrap.cc:654] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.868726 10496 tablet_bootstrap.cc:654] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:12.866988 10539 ts_tablet_manager.cc:1939] T 75a29540b579452681fc8579b6317dfc P 849223981c464676b5a7240b7660c629: Deleting consensus metadata
I20260925 14:58:12.869601 10495 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: No bootstrap required, opened a new log
I20260925 14:58:12.869652 10495 ts_tablet_manager.cc:1397] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:12.869791 10495 raft_consensus.cc:348] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.869858 10495 raft_consensus.cc:374] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.869876 10495 raft_consensus.cc:729] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.869876 9977 catalog_manager.cc:4841] TS 849223981c464676b5a7240b7660c629 (127.8.13.194:44315): tablet 75a29540b579452681fc8579b6317dfc (table test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]) successfully deleted
I20260925 14:58:12.869913 10495 consensus_queue.cc:260] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.869997 10495 ts_tablet_manager.cc:1428] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.870086 10537 ts_tablet_manager.cc:1918] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:12.870163 10537 log.cc:1196] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/75a29540b579452681fc8579b6317dfc
I20260925 14:58:12.870451 10496 tablet_bootstrap.cc:492] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: No bootstrap required, opened a new log
I20260925 14:58:12.870496 10496 ts_tablet_manager.cc:1397] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:12.870522 10537 ts_tablet_manager.cc:1939] T 75a29540b579452681fc8579b6317dfc P c193d040a7b746708db40851cbfa42f7: Deleting consensus metadata
I20260925 14:58:12.870639 10496 raft_consensus.cc:348] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.870700 10496 raft_consensus.cc:374] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:12.870718 10496 raft_consensus.cc:729] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849223981c464676b5a7240b7660c629, State: Initialized, Role: FOLLOWER
I20260925 14:58:12.870751 10496 consensus_queue.cc:260] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.870836 10496 ts_tablet_manager.cc:1428] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:12.870903 9978 catalog_manager.cc:4841] TS c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685): tablet 75a29540b579452681fc8579b6317dfc (table test-table [id=0d1c5fc4db1541c9824ac387b4df6e4d]) successfully deleted
I20260925 14:58:12.886675 10503 raft_consensus.cc:482] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:12.886768 10503 raft_consensus.cc:504] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.887065 10503 leader_election.cc:288] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 849223981c464676b5a7240b7660c629 (127.8.13.194:44315), c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685)
I20260925 14:58:12.890617 10543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.887236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44315 (local address 127.8.13.193:39933)
0925 14:58:12.887380 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.887391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.887486 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.887680 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.887682 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.887684 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.887745 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.887747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.888660 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.888661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.889707 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.889710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.890289 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.890290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.890379 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.890383 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.890464 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.890508 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":3,"thread_start_us":61,"threads_started":1}
I20260925 14:58:12.890621 10494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.887288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:39933 (local address 127.8.13.194:44315)
0925 14:58:12.887334 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.887336 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.887447 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:58:12.887598 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.887599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.887609 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.887667 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.887836 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.888580 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.889823 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.890206 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.890266 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.890276 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.890496 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.890553 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:12.890664 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.887419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43169 (local address 127.8.13.195:34685)
0925 14:58:12.887457 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.887459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.887468 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:12.887722 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.887723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.887731 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.887774 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:12.887917 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.888379 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.889799 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.890331 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.890403 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.890411 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.890561 (+ 150us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.890605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:58:12.890854 10544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.887240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.8.13.193:43169)
0925 14:58:12.887423 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.887432 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.887486 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.887792 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.887793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.887794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:12.887845 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.887846 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.888459 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.888460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.889718 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.889719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.890415 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.890416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.890480 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.890484 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.890734 (+ 250us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.890782 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260925 14:58:12.891036 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2dcc7d82694e46abab9151b5e48d95" candidate_uuid: "035a2c86749343efbe2083f516cc7c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629" is_pre_election: true
I20260925 14:58:12.891052 10429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2dcc7d82694e46abab9151b5e48d95" candidate_uuid: "035a2c86749343efbe2083f516cc7c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c193d040a7b746708db40851cbfa42f7" is_pre_election: true
I20260925 14:58:12.891124 10289 raft_consensus.cc:2454] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035a2c86749343efbe2083f516cc7c72 in term 0.
I20260925 14:58:12.891165 10429 raft_consensus.cc:2454] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 035a2c86749343efbe2083f516cc7c72 in term 0.
I20260925 14:58:12.891322 10081 leader_election.cc:302] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, 849223981c464676b5a7240b7660c629; no voters:
I20260925 14:58:12.891453 10503 raft_consensus.cc:2781] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:12.891487 10503 raft_consensus.cc:482] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:12.891510 10503 raft_consensus.cc:3037] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.892166 10503 raft_consensus.cc:504] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.892288 10503 leader_election.cc:288] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [CANDIDATE]: Term 1 election: Requested vote from peers 849223981c464676b5a7240b7660c629 (127.8.13.194:44315), c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685)
I20260925 14:58:12.892467 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2dcc7d82694e46abab9151b5e48d95" candidate_uuid: "035a2c86749343efbe2083f516cc7c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629"
I20260925 14:58:12.892541 10289 raft_consensus.cc:3037] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.892584 10429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b2dcc7d82694e46abab9151b5e48d95" candidate_uuid: "035a2c86749343efbe2083f516cc7c72" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c193d040a7b746708db40851cbfa42f7"
I20260925 14:58:12.892652 10429 raft_consensus.cc:3037] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:12.893096 10289 raft_consensus.cc:2454] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035a2c86749343efbe2083f516cc7c72 in term 1.
I20260925 14:58:12.893256 10081 leader_election.cc:302] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, 849223981c464676b5a7240b7660c629; no voters:
I20260925 14:58:12.893340 10503 raft_consensus.cc:2781] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:12.893378 10429 raft_consensus.cc:2454] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 035a2c86749343efbe2083f516cc7c72 in term 1.
I20260925 14:58:12.893456 10503 raft_consensus.cc:686] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 1 LEADER]: Becoming Leader. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Running, Role: LEADER
I20260925 14:58:12.893514 10503 consensus_queue.cc:237] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:12.894209 9991 catalog_manager.cc:5495] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 reported cstate change: term changed from 0 to 1, leader changed from <none> to 035a2c86749343efbe2083f516cc7c72 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "035a2c86749343efbe2083f516cc7c72" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:12.900616 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.898393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32946 (local address 127.8.13.193:44077)
0925 14:58:12.898426 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.898428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.898436 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:12.898499 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.898500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.898515 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.898567 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.898674 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.899136 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.899885 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.900086 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.900140 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.900298 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.900398 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.900457 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.900511 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.900533 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:58:12.905634 10493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.903300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:46469 (local address 127.8.13.195:34685)
0925 14:58:12.903341 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.903343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.903475 (+ 132us) server_negotiation.cc:407] Connection header received
0925 14:58:12.903545 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.903546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.903557 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.903599 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.903754 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.904223 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.904980 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.905187 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.905241 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.905399 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.905432 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.905471 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.905537 (+ 66us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.905555 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:12.905678 10549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.903228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.8.13.193:46469)
0925 14:58:12.903409 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.903419 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.903475 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.903610 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.903611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.903612 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:12.903696 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.903698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.904292 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.904293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.904900 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.904902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.905252 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.905253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.905316 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.905329 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.905481 (+ 152us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.905482 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.905531 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.905574 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":83,"threads_started":1}
I20260925 14:58:12.908283 10289 raft_consensus.cc:1261] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Refusing update from remote peer 035a2c86749343efbe2083f516cc7c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.908401 10429 raft_consensus.cc:1261] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Refusing update from remote peer 035a2c86749343efbe2083f516cc7c72: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:12.908581 10545 consensus_queue.cc:1035] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.908723 10503 consensus_queue.cc:1035] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:12.910470 10520 mvcc.cc:205] Tried to move back new op lower bound from 7333266607750742016 to 7333266607692374016. Current Snapshot: MvccSnapshot[applied={T|T < 7333266607750742016}]
I20260925 14:58:12.916699 10491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.913892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:42069 (local address 127.8.13.193:44077)
0925 14:58:12.914001 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:12.914003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:12.914010 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:12.914028 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:12.914029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:12.914041 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:12.914086 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:12.914214 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.914833 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.915938 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.916227 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.916306 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.916494 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.916529 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.916568 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:12.916621 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:58:12.916640 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260925 14:58:12.916746 10481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:12.913856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:44077 (local address 127.8.13.195:42069)
0925 14:58:12.913900 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:12.913911 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:12.913945 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:12.914103 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:12.914104 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:12.914105 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:12.914149 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.914151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.914912 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.914913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.915850 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:12.915854 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:12.916353 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:12.916354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:12.916418 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:12.916424 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:12.916570 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:12.916571 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:12.916618 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:12.916668 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 14:58:12.917084 10522 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:58:14.442028 9990 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:14.442133 9990 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34108:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:14.442960 9990 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Sending DeleteTablet for 3 replicas of tablet 4b2dcc7d82694e46abab9151b5e48d95
I20260925 14:58:14.443197 10269 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:34982
I20260925 14:58:14.443212 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:52390
I20260925 14:58:14.443210 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet 4b2dcc7d82694e46abab9151b5e48d95 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:32942
I20260925 14:58:14.443424 10578 tablet_replica.cc:330] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: stopping tablet replica
I20260925 14:58:14.443683 10578 raft_consensus.cc:2229] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:14.443734 10578 raft_consensus.cc:2258] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.443948 10578 ts_tablet_manager.cc:1905] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.445542 10578 ts_tablet_manager.cc:1918] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:14.445595 10578 log.cc:1196] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/4b2dcc7d82694e46abab9151b5e48d95
I20260925 14:58:14.445686 10577 tablet_replica.cc:330] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: stopping tablet replica
I20260925 14:58:14.445752 10577 raft_consensus.cc:2229] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:14.445782 10577 raft_consensus.cc:2258] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.445783 10578 ts_tablet_manager.cc:1939] T 4b2dcc7d82694e46abab9151b5e48d95 P 849223981c464676b5a7240b7660c629: Deleting consensus metadata
I20260925 14:58:14.445886 10576 tablet_replica.cc:330] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: stopping tablet replica
I20260925 14:58:14.445979 10576 raft_consensus.cc:2229] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:14.446017 10577 ts_tablet_manager.cc:1905] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.446089 10576 raft_consensus.cc:2258] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.446106 9977 catalog_manager.cc:4841] TS 849223981c464676b5a7240b7660c629 (127.8.13.194:44315): tablet 4b2dcc7d82694e46abab9151b5e48d95 (table test-table [id=887e971f163c42bc86ccf6aec14a79e6]) successfully deleted
I20260925 14:58:14.446291 10576 ts_tablet_manager.cc:1905] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.447615 10577 ts_tablet_manager.cc:1918] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:14.447750 10577 log.cc:1196] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/4b2dcc7d82694e46abab9151b5e48d95
I20260925 14:58:14.447801 10576 ts_tablet_manager.cc:1918] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:14.447871 10576 log.cc:1196] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/4b2dcc7d82694e46abab9151b5e48d95
I20260925 14:58:14.447963 10577 ts_tablet_manager.cc:1939] T 4b2dcc7d82694e46abab9151b5e48d95 P c193d040a7b746708db40851cbfa42f7: Deleting consensus metadata
I20260925 14:58:14.448127 10576 ts_tablet_manager.cc:1939] T 4b2dcc7d82694e46abab9151b5e48d95 P 035a2c86749343efbe2083f516cc7c72: Deleting consensus metadata
I20260925 14:58:14.448259 9978 catalog_manager.cc:4841] TS c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685): tablet 4b2dcc7d82694e46abab9151b5e48d95 (table test-table [id=887e971f163c42bc86ccf6aec14a79e6]) successfully deleted
I20260925 14:58:14.448431 9978 catalog_manager.cc:4841] TS 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077): tablet 4b2dcc7d82694e46abab9151b5e48d95 (table test-table [id=887e971f163c42bc86ccf6aec14a79e6]) successfully deleted
Sep 25 14:58:14 dist-test-slave-15hz krb5kdc[9947](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:14 dist-test-slave-15hz krb5kdc[9947](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:14.471043 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.459057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.8.13.254:33389)
0925 14:58:14.459178 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.459180 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.459235 (+ 55us) server_negotiation.cc:407] Connection header received
0925 14:58:14.459299 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.459300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.459311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.459368 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:14.459937 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.460487 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.461272 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.461459 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.461501 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.468808 (+ 7307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:14.468810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:14.468810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:14.468819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:14.470027 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.470349 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.470349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.470350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.470396 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:14.470620 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:14.470620 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:14.470621 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:14.470725 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:14.470794 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.470905 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.470969 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260925 14:58:14.472698 9990 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:14.472888 9990 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:14.474282 10129 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f3e84b91d4b4da362181725211de5 (DEFAULT_TABLE table=test-table [id=0598f682b15b47afb7343edacbf80b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.474296 10409 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f3e84b91d4b4da362181725211de5 (DEFAULT_TABLE table=test-table [id=0598f682b15b47afb7343edacbf80b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.474303 10269 tablet_service.cc:1467] Processing CreateTablet for tablet ea0f3e84b91d4b4da362181725211de5 (DEFAULT_TABLE table=test-table [id=0598f682b15b47afb7343edacbf80b6d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.474422 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0f3e84b91d4b4da362181725211de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.474422 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0f3e84b91d4b4da362181725211de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.474435 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea0f3e84b91d4b4da362181725211de5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.475564 10587 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Bootstrap starting.
I20260925 14:58:14.475821 10588 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Bootstrap starting.
I20260925 14:58:14.475975 10589 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Bootstrap starting.
I20260925 14:58:14.475993 10587 tablet_bootstrap.cc:654] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.476171 10588 tablet_bootstrap.cc:654] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.476409 10589 tablet_bootstrap.cc:654] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.476620 10587 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: No bootstrap required, opened a new log
I20260925 14:58:14.476683 10587 ts_tablet_manager.cc:1397] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:14.476824 10587 raft_consensus.cc:348] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.476915 10587 raft_consensus.cc:374] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.476938 10587 raft_consensus.cc:729] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849223981c464676b5a7240b7660c629, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.477012 10587 consensus_queue.cc:260] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.477103 10589 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: No bootstrap required, opened a new log
I20260925 14:58:14.477159 10589 ts_tablet_manager.cc:1397] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:14.477195 10587 ts_tablet_manager.cc:1428] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:14.477296 10589 raft_consensus.cc:348] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.477391 10589 raft_consensus.cc:374] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.477438 10589 raft_consensus.cc:729] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.477505 10589 consensus_queue.cc:260] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.477620 10588 tablet_bootstrap.cc:492] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: No bootstrap required, opened a new log
I20260925 14:58:14.477677 10588 ts_tablet_manager.cc:1397] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:14.477754 10589 ts_tablet_manager.cc:1428] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:14.477790 10588 raft_consensus.cc:348] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.477869 10588 raft_consensus.cc:374] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.477900 10588 raft_consensus.cc:729] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.477947 10588 consensus_queue.cc:260] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.478049 10588 ts_tablet_manager.cc:1428] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.733016 10593 raft_consensus.cc:482] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:14.733115 10593 raft_consensus.cc:504] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.733445 10593 leader_election.cc:288] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685)
I20260925 14:58:14.737669 10595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.733736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:57187 (local address 127.8.13.193:44077)
0925 14:58:14.733883 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.733885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.733899 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:58:14.733998 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.733999 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.734011 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.734102 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:14.734272 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.735070 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.736657 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.737263 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.737332 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.737341 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.737543 (+ 202us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.737585 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 14:58:14.737669 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.734011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:58973 (local address 127.8.13.195:34685)
0925 14:58:14.734174 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.734175 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.734183 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:14.734205 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.734205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.734214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.734279 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:14.734406 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.735093 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.736656 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.737251 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.737331 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.737341 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.737547 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.737589 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260925 14:58:14.737718 10594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.733661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:44077 (local address 127.8.13.194:57187)
0925 14:58:14.733842 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.733855 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.733918 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.734127 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.734128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.734130 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:14.734199 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.734200 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.735209 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.735210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.736562 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.736566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.737365 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.737366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.737456 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.737463 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.737526 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.737580 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"mutex_wait_us":9,"thread_start_us":103,"threads_started":1}
I20260925 14:58:14.737989 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.733949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.8.13.194:58973)
0925 14:58:14.734068 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.734076 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.734121 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.734297 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.734298 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.734300 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:14.734345 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.734346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.735209 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.735210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.736562 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.736566 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.737378 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.737379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.737456 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.737464 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.737534 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.737580 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":120,"thread_start_us":68,"threads_started":1}
I20260925 14:58:14.738065 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f3e84b91d4b4da362181725211de5" candidate_uuid: "849223981c464676b5a7240b7660c629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72" is_pre_election: true
I20260925 14:58:14.738176 10148 raft_consensus.cc:2454] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849223981c464676b5a7240b7660c629 in term 0.
I20260925 14:58:14.738273 10429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f3e84b91d4b4da362181725211de5" candidate_uuid: "849223981c464676b5a7240b7660c629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c193d040a7b746708db40851cbfa42f7" is_pre_election: true
I20260925 14:58:14.738364 10429 raft_consensus.cc:2454] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 849223981c464676b5a7240b7660c629 in term 0.
I20260925 14:58:14.738345 10222 leader_election.cc:302] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, 849223981c464676b5a7240b7660c629; no voters:
I20260925 14:58:14.738516 10593 raft_consensus.cc:2781] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:14.738560 10593 raft_consensus.cc:482] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:14.738582 10593 raft_consensus.cc:3037] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:14.739221 10593 raft_consensus.cc:504] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.739336 10593 leader_election.cc:288] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [CANDIDATE]: Term 1 election: Requested vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685)
I20260925 14:58:14.739511 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f3e84b91d4b4da362181725211de5" candidate_uuid: "849223981c464676b5a7240b7660c629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72"
I20260925 14:58:14.739560 10429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea0f3e84b91d4b4da362181725211de5" candidate_uuid: "849223981c464676b5a7240b7660c629" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c193d040a7b746708db40851cbfa42f7"
I20260925 14:58:14.739629 10429 raft_consensus.cc:3037] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:14.739631 10148 raft_consensus.cc:3037] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:14.740200 10429 raft_consensus.cc:2454] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849223981c464676b5a7240b7660c629 in term 1.
I20260925 14:58:14.740298 10148 raft_consensus.cc:2454] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 849223981c464676b5a7240b7660c629 in term 1.
I20260925 14:58:14.740352 10222 leader_election.cc:302] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849223981c464676b5a7240b7660c629, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:14.740453 10593 raft_consensus.cc:2781] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:14.740602 10593 raft_consensus.cc:686] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 1 LEADER]: Becoming Leader. State: Replica: 849223981c464676b5a7240b7660c629, State: Running, Role: LEADER
I20260925 14:58:14.740666 10593 consensus_queue.cc:237] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.741403 9990 catalog_manager.cc:5495] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 reported cstate change: term changed from 0 to 1, leader changed from <none> to 849223981c464676b5a7240b7660c629 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "849223981c464676b5a7240b7660c629" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:14.763942 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.761614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34994 (local address 127.8.13.194:44315)
0925 14:58:14.761748 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.761750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.761755 (+ 5us) server_negotiation.cc:407] Connection header received
0925 14:58:14.761787 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.761788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.761806 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.761866 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:14.762036 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.762541 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.763272 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.763468 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.763511 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.763668 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:14.763774 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:14.763825 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.763854 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.763870 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260925 14:58:14.768571 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.766520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:55011 (local address 127.8.13.195:34685)
0925 14:58:14.766572 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.766573 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.766580 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:14.766629 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.766630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.766639 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.766681 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:14.766794 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.767245 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.768006 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.768216 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.768264 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.768388 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:14.768412 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:14.768446 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.768488 (+ 42us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.768506 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:58:14.768600 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.766347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34685 (local address 127.8.13.194:55011)
0925 14:58:14.766499 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.766511 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.766567 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.766694 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.766695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.766696 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:14.766751 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.766752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.767301 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.767301 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.767938 (+ 637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.767940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.768273 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.768274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.768317 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.768330 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:14.768456 (+ 126us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:14.768457 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.768486 (+ 29us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.768520 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260925 14:58:14.770470 10429 raft_consensus.cc:1261] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Refusing update from remote peer 849223981c464676b5a7240b7660c629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:14.770519 10148 raft_consensus.cc:1261] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Refusing update from remote peer 849223981c464676b5a7240b7660c629: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:14.770745 10598 consensus_queue.cc:1035] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:14.770852 10593 consensus_queue.cc:1035] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:14.778054 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.775683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:49647 (local address 127.8.13.194:44315)
0925 14:58:14.775723 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:14.775725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:14.775852 (+ 127us) server_negotiation.cc:407] Connection header received
0925 14:58:14.775906 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:14.775907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:14.775916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:14.775954 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:14.776036 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.776511 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.777279 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.777560 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.777625 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.777825 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:14.777862 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:14.777904 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:14.777956 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:58:14.777981 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:14.778095 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:14.775649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44315 (local address 127.8.13.195:49647)
0925 14:58:14.775783 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:14.775794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:14.775853 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:14.775957 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:14.775958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:14.775959 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:14.776003 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.776004 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.776565 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.776566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.777179 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:14.777182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:14.777661 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:14.777682 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:14.777754 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:14.777760 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:14.777914 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:14.777916 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:14.777955 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 14:58:14.778026 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":95,"threads_started":1}
I20260925 14:58:14.784682 9992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:14.784829 9992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:14.785501 9992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Sending DeleteTablet for 3 replicas of tablet ea0f3e84b91d4b4da362181725211de5
I20260925 14:58:14.785780 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f3e84b91d4b4da362181725211de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:32942
I20260925 14:58:14.785830 10269 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f3e84b91d4b4da362181725211de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:34982
I20260925 14:58:14.785882 10576 tablet_replica.cc:330] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: stopping tablet replica
I20260925 14:58:14.785928 10576 raft_consensus.cc:2229] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:14.785923 10578 tablet_replica.cc:330] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: stopping tablet replica
I20260925 14:58:14.785964 10576 raft_consensus.cc:2258] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.785974 10578 raft_consensus.cc:2229] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:14.786062 10578 raft_consensus.cc:2258] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.786064 9992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:14.786216 9992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:14.786401 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet ea0f3e84b91d4b4da362181725211de5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:14 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:52390
I20260925 14:58:14.786422 10578 ts_tablet_manager.cc:1905] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.786633 10576 ts_tablet_manager.cc:1905] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.787459 10129 tablet_service.cc:1467] Processing CreateTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a (DEFAULT_TABLE table=test-table [id=3d5e8a1c8d4749b395ac84ea799fca43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.787596 10129 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80e0a31d3d3437994c4d772b73dbb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.788007 10577 tablet_replica.cc:330] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: stopping tablet replica
I20260925 14:58:14.788089 10577 raft_consensus.cc:2229] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:14.788144 10577 raft_consensus.cc:2258] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:14.788173 10576 ts_tablet_manager.cc:1918] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:14.788220 10576 log.cc:1196] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/ea0f3e84b91d4b4da362181725211de5
I20260925 14:58:14.788286 10578 ts_tablet_manager.cc:1918] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:14.788337 10578 log.cc:1196] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/ea0f3e84b91d4b4da362181725211de5
I20260925 14:58:14.788411 10576 ts_tablet_manager.cc:1939] T ea0f3e84b91d4b4da362181725211de5 P 035a2c86749343efbe2083f516cc7c72: Deleting consensus metadata
I20260925 14:58:14.788508 10578 ts_tablet_manager.cc:1939] T ea0f3e84b91d4b4da362181725211de5 P 849223981c464676b5a7240b7660c629: Deleting consensus metadata
I20260925 14:58:14.788566 10577 ts_tablet_manager.cc:1905] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:14.788707 10269 tablet_service.cc:1467] Processing CreateTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a (DEFAULT_TABLE table=test-table [id=3d5e8a1c8d4749b395ac84ea799fca43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.788763 10588 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: Bootstrap starting.
I20260925 14:58:14.788794 10269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80e0a31d3d3437994c4d772b73dbb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.788800 10409 tablet_service.cc:1467] Processing CreateTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a (DEFAULT_TABLE table=test-table [id=3d5e8a1c8d4749b395ac84ea799fca43]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:14.788887 10409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d80e0a31d3d3437994c4d772b73dbb9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:14.789034 10588 tablet_bootstrap.cc:654] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.789779 10587 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: Bootstrap starting.
I20260925 14:58:14.789856 9977 catalog_manager.cc:4841] TS 849223981c464676b5a7240b7660c629 (127.8.13.194:44315): tablet ea0f3e84b91d4b4da362181725211de5 (table test-table [id=0598f682b15b47afb7343edacbf80b6d]) successfully deleted
I20260925 14:58:14.789928 10589 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Bootstrap starting.
I20260925 14:58:14.790145 10587 tablet_bootstrap.cc:654] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.790232 10589 tablet_bootstrap.cc:654] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:14.790341 10577 ts_tablet_manager.cc:1918] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:14.790387 10577 log.cc:1196] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/ea0f3e84b91d4b4da362181725211de5
I20260925 14:58:14.790583 10577 ts_tablet_manager.cc:1939] T ea0f3e84b91d4b4da362181725211de5 P c193d040a7b746708db40851cbfa42f7: Deleting consensus metadata
I20260925 14:58:14.790874 9978 catalog_manager.cc:4841] TS c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685): tablet ea0f3e84b91d4b4da362181725211de5 (table test-table [id=0598f682b15b47afb7343edacbf80b6d]) successfully deleted
I20260925 14:58:14.790947 9978 catalog_manager.cc:4841] TS 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077): tablet ea0f3e84b91d4b4da362181725211de5 (table test-table [id=0598f682b15b47afb7343edacbf80b6d]) successfully deleted
I20260925 14:58:14.791220 10589 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: No bootstrap required, opened a new log
I20260925 14:58:14.791268 10589 ts_tablet_manager.cc:1397] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:14.791370 10589 raft_consensus.cc:348] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.791419 10588 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: No bootstrap required, opened a new log
I20260925 14:58:14.791432 10589 raft_consensus.cc:374] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.791452 10589 raft_consensus.cc:729] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.791460 10588 ts_tablet_manager.cc:1397] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:14.791509 10589 consensus_queue.cc:260] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.791594 10588 raft_consensus.cc:348] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.791631 10589 ts_tablet_manager.cc:1428] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.791658 10588 raft_consensus.cc:374] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.791678 10588 raft_consensus.cc:729] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 035a2c86749343efbe2083f516cc7c72, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.791715 10588 consensus_queue.cc:260] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.791802 10588 ts_tablet_manager.cc:1428] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:14.792282 10587 tablet_bootstrap.cc:492] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: No bootstrap required, opened a new log
I20260925 14:58:14.792336 10587 ts_tablet_manager.cc:1397] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:14.792490 10587 raft_consensus.cc:348] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.792552 10587 raft_consensus.cc:374] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:14.792565 10587 raft_consensus.cc:729] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849223981c464676b5a7240b7660c629, State: Initialized, Role: FOLLOWER
I20260925 14:58:14.792594 10587 consensus_queue.cc:260] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:14.792706 10587 ts_tablet_manager.cc:1428] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:15.157994 10531 raft_consensus.cc:482] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:15.158099 10531 raft_consensus.cc:504] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:15.158262 10531 leader_election.cc:288] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:15.158496 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80e0a31d3d3437994c4d772b73dbb9a" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72" is_pre_election: true
I20260925 14:58:15.158504 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80e0a31d3d3437994c4d772b73dbb9a" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629" is_pre_election: true
I20260925 14:58:15.158591 10289 raft_consensus.cc:2454] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:15.158594 10148 raft_consensus.cc:2454] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 0.
I20260925 14:58:15.158751 10362 leader_election.cc:302] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 035a2c86749343efbe2083f516cc7c72, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:15.158879 10531 raft_consensus.cc:2781] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:15.158931 10531 raft_consensus.cc:482] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:15.158949 10531 raft_consensus.cc:3037] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:15.159688 10531 raft_consensus.cc:504] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:15.159798 10531 leader_election.cc:288] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Requested vote from peers 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077), 849223981c464676b5a7240b7660c629 (127.8.13.194:44315)
I20260925 14:58:15.159941 10289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80e0a31d3d3437994c4d772b73dbb9a" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "849223981c464676b5a7240b7660c629"
I20260925 14:58:15.159961 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d80e0a31d3d3437994c4d772b73dbb9a" candidate_uuid: "c193d040a7b746708db40851cbfa42f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "035a2c86749343efbe2083f516cc7c72"
I20260925 14:58:15.160005 10289 raft_consensus.cc:3037] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:15.160032 10148 raft_consensus.cc:3037] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:15.160576 10148 raft_consensus.cc:2454] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:15.160591 10289 raft_consensus.cc:2454] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c193d040a7b746708db40851cbfa42f7 in term 1.
I20260925 14:58:15.160728 10361 leader_election.cc:302] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 849223981c464676b5a7240b7660c629, c193d040a7b746708db40851cbfa42f7; no voters:
I20260925 14:58:15.160836 10531 raft_consensus.cc:2781] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:15.160892 10531 raft_consensus.cc:686] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 1 LEADER]: Becoming Leader. State: Replica: c193d040a7b746708db40851cbfa42f7, State: Running, Role: LEADER
I20260925 14:58:15.160939 10531 consensus_queue.cc:237] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } }
I20260925 14:58:15.161512 9992 catalog_manager.cc:5495] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 reported cstate change: term changed from 0 to 1, leader changed from <none> to c193d040a7b746708db40851cbfa42f7 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "c193d040a7b746708db40851cbfa42f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c193d040a7b746708db40851cbfa42f7" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34685 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:15.236115 10597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:15.233969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52400 (local address 127.8.13.195:34685)
0925 14:58:15.234013 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:15.234014 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:15.234022 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:15.234092 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:15.234092 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:15.234102 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:15.234139 (+ 37us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:15.234269 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.234749 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.235464 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:15.235684 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:15.235741 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:15.235874 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:15.235909 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:15.235949 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:15.236008 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 14:58:15.236030 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:15.238511 10289 raft_consensus.cc:1261] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:15.238539 10148 raft_consensus.cc:1261] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Refusing update from remote peer c193d040a7b746708db40851cbfa42f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:15.238751 10556 consensus_queue.cc:1035] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "035a2c86749343efbe2083f516cc7c72" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 44077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:15.238869 10561 consensus_queue.cc:1035] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "849223981c464676b5a7240b7660c629" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:15.243222 10554 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260925 14:58:16.761206 9992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:16.761304 9992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34128:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:16.762048 9992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 32c428adf3cc433eb6fa672c19f5b4eb: Sending DeleteTablet for 3 replicas of tablet d80e0a31d3d3437994c4d772b73dbb9a
I20260925 14:58:16.762296 10129 tablet_service.cc:1514] Processing DeleteTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:32942
I20260925 14:58:16.762297 10409 tablet_service.cc:1514] Processing DeleteTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:52390
I20260925 14:58:16.762362 10269 tablet_service.cc:1514] Processing DeleteTablet for tablet d80e0a31d3d3437994c4d772b73dbb9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:16 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:34982
I20260925 14:58:16.762517 10638 tablet_replica.cc:330] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72: stopping tablet replica
I20260925 14:58:16.762580 10638 raft_consensus.cc:2229] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:16.762619 10638 raft_consensus.cc:2258] T d80e0a31d3d3437994c4d772b73dbb9a P 035a2c86749343efbe2083f516cc7c72 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.762667 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10064
I20260925 14:58:16.762713 10637 tablet_replica.cc:330] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: stopping tablet replica
I20260925 14:58:16.762800 10637 raft_consensus.cc:2229] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:16.762868 10637 raft_consensus.cc:2258] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.763161 10637 ts_tablet_manager.cc:1905] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:16.765128 10637 ts_tablet_manager.cc:1918] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260925 14:58:16.765190 10637 log.cc:1196] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/d80e0a31d3d3437994c4d772b73dbb9a
I20260925 14:58:16.765417 10637 ts_tablet_manager.cc:1939] T d80e0a31d3d3437994c4d772b73dbb9a P c193d040a7b746708db40851cbfa42f7: Deleting consensus metadata
I20260925 14:58:16.765717 9978 catalog_manager.cc:4841] TS c193d040a7b746708db40851cbfa42f7 (127.8.13.195:34685): tablet d80e0a31d3d3437994c4d772b73dbb9a (table test-table [id=3d5e8a1c8d4749b395ac84ea799fca43]) successfully deleted
I20260925 14:58:16.770910 10639 tablet_replica.cc:330] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629: stopping tablet replica
W20260925 14:58:16.770934 9978 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:44077 (error 108)
I20260925 14:58:16.770992 10639 raft_consensus.cc:2229] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:16.771040 10639 raft_consensus.cc:2258] T d80e0a31d3d3437994c4d772b73dbb9a P 849223981c464676b5a7240b7660c629 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:16.771057 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10204
I20260925 14:58:16.772207 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.771903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:44077 (local address 127.0.0.1:46054)
0925 14:58:16.772110 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.772157 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:44077: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":64,"threads_started":1}
W20260925 14:58:16.772833 9978 catalog_manager.cc:4584] TS 035a2c86749343efbe2083f516cc7c72 (127.8.13.193:44077): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d80e0a31d3d3437994c4d772b73dbb9a: Network error: Client connection negotiation failed: client connection to 127.8.13.193:44077: connect: Connection refused (error 111)
I20260925 14:58:16.776453 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10344
I20260925 14:58:16.776494 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:16.776361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44315 (local address 127.0.0.1:53900)
0925 14:58:16.776436 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:16.776462 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:44315: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260925 14:58:16.781191 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 9960
2026-09-25T14:58:16Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5972 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:16 dist-test-slave-15hz krb5kdc[10644](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:16 dist-test-slave-15hz krb5kdc[10644](info): set up 2 sockets
Sep 25 14:58:16 dist-test-slave-15hz krb5kdc[10644](info): commencing operation
krb5kdc: starting...
W20260925 14:58:17.731184 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:17 dist-test-slave-15hz krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:17.845860 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34657
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:34657
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:17.924491 10657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:17.924768 10657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:17.924808 10657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:17.926074 10657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:17.926129 10657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:17.926148 10657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:17.926165 10657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:17.926182 10657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:17.927855 10657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:34657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:17.928133 10657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:17.928400 10657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:17.933305 10662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:17.933375 10657 server_base.cc:1034] running on GCE node
W20260925 14:58:17.933297 10663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:17.933297 10665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:17.933801 10657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:17.934084 10657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:17.935237 10657 hybrid_clock.cc:648] HybridClock initialized: now 1790348297935223 us; error 32 us; skew 500 ppm
Sep 25 14:58:17 dist-test-slave-15hz krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348297, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:17.937836 10657 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:17.938293 10657 webserver.cc:460] Webserver started at http://127.8.13.254:45865/ using document root <none> and password file <none>
I20260925 14:58:17.938521 10657 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:17.938572 10657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:17.938720 10657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:17.939868 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8"
format_stamp: "Formatted at 2026-09-25 14:58:17 on dist-test-slave-15hz"
server_key: "a4540e1c10dce3aaf9104ae300b49c4e"
server_key_iv: "105f8c347edd5c0a80dc56e8fd7a12f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:17.940213 10657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8"
format_stamp: "Formatted at 2026-09-25 14:58:17 on dist-test-slave-15hz"
server_key: "a4540e1c10dce3aaf9104ae300b49c4e"
server_key_iv: "105f8c347edd5c0a80dc56e8fd7a12f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:17.941470 10657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:17.942167 10672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:17.942333 10657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:17.942418 10657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8"
format_stamp: "Formatted at 2026-09-25 14:58:17 on dist-test-slave-15hz"
server_key: "a4540e1c10dce3aaf9104ae300b49c4e"
server_key_iv: "105f8c347edd5c0a80dc56e8fd7a12f4"
server_key_version: "encryptionkey@0"
I20260925 14:58:17.942485 10657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:17.952880 10657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:17.955152 10657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:17.955296 10657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:17.958998 10657 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:34657
I20260925 14:58:17.959043 10724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:34657 every 8 connection(s)
I20260925 14:58:17.959385 10657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:17.960222 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10657
I20260925 14:58:17.960263 10725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:17.960302 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:17.960457 8247 external_mini_cluster.cc:1428] Setting key 8e7e24363af6c980d33a60c92a9eb664
I20260925 14:58:17.963487 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8: Bootstrap starting.
I20260925 14:58:17.964184 10725 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:17.964442 10725 log.cc:824] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:17.965113 10725 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8: No bootstrap required, opened a new log
Sep 25 14:58:17 dist-test-slave-15hz krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348297, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:17.967134 10725 raft_consensus.cc:348] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } }
I20260925 14:58:17.967305 10725 raft_consensus.cc:374] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:17.967353 10725 raft_consensus.cc:729] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffbea3adfab44bf7ab2abf9d0e80b0d8, State: Initialized, Role: FOLLOWER
I20260925 14:58:17.967486 10725 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } }
I20260925 14:58:17.967603 10725 raft_consensus.cc:388] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:17.967641 10725 raft_consensus.cc:482] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:17.967690 10725 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:17.968300 10725 raft_consensus.cc:504] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } }
I20260925 14:58:17.968401 10725 leader_election.cc:302] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffbea3adfab44bf7ab2abf9d0e80b0d8; no voters:
I20260925 14:58:17.968555 10725 leader_election.cc:288] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:17.968605 10730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:17.968793 10730 raft_consensus.cc:686] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [term 1 LEADER]: Becoming Leader. State: Replica: ffbea3adfab44bf7ab2abf9d0e80b0d8, State: Running, Role: LEADER
I20260925 14:58:17.968820 10725 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:17.968971 10730 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } }
I20260925 14:58:17.969336 10732 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffbea3adfab44bf7ab2abf9d0e80b0d8. Latest consensus state: current_term: 1 leader_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } } }
I20260925 14:58:17.969448 10732 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:17.969636 10739 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:17.970347 10739 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:17.970662 10731 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffbea3adfab44bf7ab2abf9d0e80b0d8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34657 } } }
I20260925 14:58:17.970775 10731 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:17.971674 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:17.961032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37148 (local address 127.8.13.254:34657)
0925 14:58:17.961256 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:17.961265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:17.961298 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:17.961745 (+ 447us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:17.961755 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:17.961824 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:17.961977 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:17.962501 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.963942 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.964759 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:17.965173 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:17.965281 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:17.967830 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:17.967835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:17.967835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:17.967845 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:17.969745 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.970150 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.970154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.970154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.970258 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:17.970456 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:17.970457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:17.970458 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:17.970761 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:17.970888 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:17.971322 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 14:58:17.971452 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":111,"threads_started":1}
I20260925 14:58:17.972071 10739 catalog_manager.cc:1347] Generated new cluster ID: e7f37c9d1e94469581fcb9308ed0f37d
I20260925 14:58:17.972121 10739 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:17.988469 10739 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:17.989233 10739 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:18.000151 10739 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ffbea3adfab44bf7ab2abf9d0e80b0d8: Generated new TSK 0
I20260925 14:58:18.000363 10739 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:18.032438 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:18.121950 10751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:18.122167 10751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:18.122191 10751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:18.123377 10751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:18.123425 10751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:18.123450 10751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:18.124980 10751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:18.125221 10751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:18.125447 10751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:18.130226 10757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.130240 10759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.130242 10756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:18.130517 10751 server_base.cc:1034] running on GCE node
I20260925 14:58:18.130712 10751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:18.130955 10751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:18.132100 10751 hybrid_clock.cc:648] HybridClock initialized: now 1790348298132081 us; error 27 us; skew 500 ppm
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:18.134459 10751 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:18.134918 10751 webserver.cc:460] Webserver started at http://127.8.13.193:41151/ using document root <none> and password file <none>
I20260925 14:58:18.135139 10751 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:18.135188 10751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:18.135314 10751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:18.136394 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "7aa66ec4f42940d1b78e5b705e54bbde"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "d7dc1c39c96f2d16aed303fe118028bf"
server_key_iv: "0de8c1bdc46588ed9e662890b5bba86a"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.136727 10751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "7aa66ec4f42940d1b78e5b705e54bbde"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "d7dc1c39c96f2d16aed303fe118028bf"
server_key_iv: "0de8c1bdc46588ed9e662890b5bba86a"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.137919 10751 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:58:18.138581 10766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.138743 10751 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:18.138823 10751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "7aa66ec4f42940d1b78e5b705e54bbde"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "d7dc1c39c96f2d16aed303fe118028bf"
server_key_iv: "0de8c1bdc46588ed9e662890b5bba86a"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.138890 10751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:18.152313 10751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:18.154428 10751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:18.154565 10751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:18.154770 10751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:18.155050 10751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:18.155081 10751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.155117 10751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:18.155141 10751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.160533 10751 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:38497
I20260925 14:58:18.160600 10879 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:38497 every 8 connection(s)
I20260925 14:58:18.160880 10751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:18.167869 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10751
I20260925 14:58:18.167943 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:18.168097 8247 external_mini_cluster.cc:1428] Setting key fdf63613e345073c84f929d43baa0295
I20260925 14:58:18.169497 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.161549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:36833 (local address 127.8.13.254:34657)
0925 14:58:18.161631 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:18.161634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:18.161771 (+ 137us) server_negotiation.cc:407] Connection header received
0925 14:58:18.162231 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:18.162232 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:18.162241 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:18.162290 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:18.163171 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.163730 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.164673 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.164874 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.164918 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.166864 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:18.166866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:18.166866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:18.166872 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:18.168060 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.168487 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.168488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.168488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.168548 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.168784 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.168785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.168785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.168915 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:18.168969 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:18.169295 (+ 326us) server_negotiation.cc:299] Negotiation successful
0925 14:58:18.169409 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:58:18.169725 10882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.161512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:34657 (local address 127.8.13.193:36833)
0925 14:58:18.161651 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:18.161687 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:18.162151 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:18.162343 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:18.162346 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:18.162881 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:18.163073 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.163078 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.163798 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.163798 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.164606 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.164608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.164929 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.164929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.165026 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.165522 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:18.165530 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:18.166773 (+ 1243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:18.168152 (+ 1379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.168159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.168161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.168428 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.168597 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.168598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.168599 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.168720 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.168981 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:18.168988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:18.169149 (+ 161us) client_negotiation.cc:769] Sending connection context
0925 14:58:18.169259 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 14:58:18.169464 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
I20260925 14:58:18.170181 10880 heartbeater.cc:347] Connected to a master server at 127.8.13.254:34657
I20260925 14:58:18.170274 10880 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:18.170456 10880 heartbeater.cc:511] Master 127.8.13.254:34657 requested a full tablet report, sending...
I20260925 14:58:18.170959 10689 ts_manager.cc:194] Registered new tserver with Master: 7aa66ec4f42940d1b78e5b705e54bbde (127.8.13.193:38497)
I20260925 14:58:18.171819 10689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:36833
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:18.190385 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:18.271430 10885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:18.271710 10885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:18.271737 10885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:18.272970 10885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:18.273012 10885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:18.273041 10885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:18.274538 10885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10885
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:18.274744 10885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:18.274936 10885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:18.279791 10890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.279791 10891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.279847 10893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:18.279978 10885 server_base.cc:1034] running on GCE node
I20260925 14:58:18.280222 10885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:18.280455 10885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:18.281625 10885 hybrid_clock.cc:648] HybridClock initialized: now 1790348298281607 us; error 31 us; skew 500 ppm
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:18.284181 10885 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:18.284760 10885 webserver.cc:460] Webserver started at http://127.8.13.194:34907/ using document root <none> and password file <none>
I20260925 14:58:18.284963 10885 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:18.285005 10885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:18.285102 10885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:18.286144 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "fd6565e802ef429cac729fb6edc7e10e"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "b24cb95ecbfb3ef334e91827fd3956bc"
server_key_iv: "6c97a365a084504a2e5cd59d483803ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.286450 10885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "fd6565e802ef429cac729fb6edc7e10e"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "b24cb95ecbfb3ef334e91827fd3956bc"
server_key_iv: "6c97a365a084504a2e5cd59d483803ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.287775 10885 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:18.288458 10900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.288640 10885 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:18.288715 10885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "fd6565e802ef429cac729fb6edc7e10e"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "b24cb95ecbfb3ef334e91827fd3956bc"
server_key_iv: "6c97a365a084504a2e5cd59d483803ec"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.288772 10885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:18.299208 10885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:18.301426 10885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:18.301539 10885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:18.301736 10885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:18.301991 10885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:18.302021 10885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.302042 10885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:18.302052 10885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.307590 10885 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:39469
I20260925 14:58:18.307634 11013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:39469 every 8 connection(s)
I20260925 14:58:18.307950 10885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:18.315092 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 10885
I20260925 14:58:18.315215 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:18.315407 8247 external_mini_cluster.cc:1428] Setting key 98669374e1d114d91ec3320dd7137c96
I20260925 14:58:18.317597 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.308753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:52029 (local address 127.8.13.254:34657)
0925 14:58:18.308808 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:18.308811 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:18.309015 (+ 204us) server_negotiation.cc:407] Connection header received
0925 14:58:18.309483 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:18.309484 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:18.309494 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:18.309554 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:18.310463 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.310959 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.312068 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.312280 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.312349 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.314682 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:18.314684 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:18.314684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:18.314692 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:18.316207 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.316641 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.316642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.316642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.316696 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.316935 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.316936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.316936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.317080 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:18.317135 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:18.317423 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 14:58:18.317506 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:18.317797 11016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.308728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:34657 (local address 127.8.13.194:52029)
0925 14:58:18.308889 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:18.308933 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:18.309407 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:18.309616 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:18.309619 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:18.310181 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:18.310383 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.310388 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.311036 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.311037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.311948 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.311952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.312379 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.312380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.312498 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.313038 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:18.313047 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:18.314545 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:18.316305 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.316315 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.316317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.316565 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.316761 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.316762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.316762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.316874 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.317148 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:18.317156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:18.317300 (+ 144us) client_negotiation.cc:769] Sending connection context
0925 14:58:18.317391 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:58:18.317567 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
I20260925 14:58:18.318225 11014 heartbeater.cc:347] Connected to a master server at 127.8.13.254:34657
I20260925 14:58:18.318331 11014 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:18.318511 11014 heartbeater.cc:511] Master 127.8.13.254:34657 requested a full tablet report, sending...
I20260925 14:58:18.318917 10689 ts_manager.cc:194] Registered new tserver with Master: fd6565e802ef429cac729fb6edc7e10e (127.8.13.194:39469)
I20260925 14:58:18.319289 10689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:52029
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:18.339495 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:46883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:18.418519 11019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:18.418748 11019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:18.418777 11019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:18.420073 11019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:18.420121 11019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:18.420156 11019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:18.421661 11019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:46883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:34657
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:18.421909 11019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:18.422135 11019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:18.427103 11025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.427223 11024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:18.427096 11027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:18.427340 11019 server_base.cc:1034] running on GCE node
I20260925 14:58:18.427574 11019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:18.427834 11019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:18.428980 11019 hybrid_clock.cc:648] HybridClock initialized: now 1790348298428958 us; error 37 us; skew 500 ppm
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:18.431401 11019 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:18.431878 11019 webserver.cc:460] Webserver started at http://127.8.13.195:35375/ using document root <none> and password file <none>
I20260925 14:58:18.432086 11019 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:18.432135 11019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:18.432260 11019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:18.433275 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "703b60810d54406cb09ae3eec7b9c290"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "03db93888f8c7a3258917de6afed75fc"
server_key_iv: "1e09a2ed3390a6be65a6c8dad3596ccd"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.433637 11019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "703b60810d54406cb09ae3eec7b9c290"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "03db93888f8c7a3258917de6afed75fc"
server_key_iv: "1e09a2ed3390a6be65a6c8dad3596ccd"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.434859 11019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:18.435556 11034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.435717 11019 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:18.435801 11019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "703b60810d54406cb09ae3eec7b9c290"
format_stamp: "Formatted at 2026-09-25 14:58:18 on dist-test-slave-15hz"
server_key: "03db93888f8c7a3258917de6afed75fc"
server_key_iv: "1e09a2ed3390a6be65a6c8dad3596ccd"
server_key_version: "encryptionkey@0"
I20260925 14:58:18.435868 11019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:18.446537 11019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:18.448709 11019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:18.448851 11019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:18.449061 11019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:18.449368 11019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:18.449411 11019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.449441 11019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:18.449465 11019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:18.454835 11019 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:45321
I20260925 14:58:18.454876 11147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:45321 every 8 connection(s)
I20260925 14:58:18.455174 11019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[10644](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348298, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:18.463922 10728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.455909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:36467 (local address 127.8.13.254:34657)
0925 14:58:18.455959 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:18.455961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:18.456158 (+ 197us) server_negotiation.cc:407] Connection header received
0925 14:58:18.456632 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:18.456633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:18.456642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:18.456687 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:18.457569 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.458055 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.459002 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.459211 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.459261 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.461366 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:18.461367 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:18.461368 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:18.461375 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:18.462690 (+ 1315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.463053 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.463054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.463054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.463105 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:18.463298 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:18.463299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:18.463299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:18.463400 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:18.463447 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:18.463765 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 14:58:18.463833 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:18.464068 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11019
I20260925 14:58:18.464165 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:18.464177 11150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:18.455885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:34657 (local address 127.8.13.195:36467)
0925 14:58:18.456036 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:18.456073 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:18.456550 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:18.456751 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:18.456755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:18.457303 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:18.457499 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.457504 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.458122 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.458123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.458947 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:18.458949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:18.459275 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:18.459275 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:18.459368 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:18.459876 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:18.459885 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:18.461176 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:18.462756 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.462760 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.462762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.462960 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.463157 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:18.463158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:18.463158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:18.463250 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:18.463446 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:18.463450 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:18.463609 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:58:18.463729 (+ 120us) client_negotiation.cc:240] Negotiation successful
0925 14:58:18.463892 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":76,"threads_started":1}
I20260925 14:58:18.464361 8247 external_mini_cluster.cc:1428] Setting key 29f1b9a2a5a6501872bb57cc85c75fd6
I20260925 14:58:18.464691 11148 heartbeater.cc:347] Connected to a master server at 127.8.13.254:34657
I20260925 14:58:18.464808 11148 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:18.465001 11148 heartbeater.cc:511] Master 127.8.13.254:34657 requested a full tablet report, sending...
I20260925 14:58:18.465422 10689 ts_manager.cc:194] Registered new tserver with Master: 703b60810d54406cb09ae3eec7b9c290 (127.8.13.195:45321)
I20260925 14:58:18.465785 10689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:36467
I20260925 14:58:18.476217 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:58:18.485998 10728 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:18.485203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37152 (local address 127.8.13.254:34657)
0925 14:58:18.485279 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:18.485281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:18.485330 (+ 49us) server_negotiation.cc:407] Connection header received
0925 14:58:18.485387 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:18.485388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:18.485397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:18.485445 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:18.485926 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37152: BlockingRecv error: recv got EOF from 127.0.0.1:37152 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:58:18.486961 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10751
I20260925 14:58:18.491010 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10885
I20260925 14:58:18.494683 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11019
I20260925 14:58:18.498193 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 10657
2026-09-25T14:58:18Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1716 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[11160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[11160](info): set up 2 sockets
Sep 25 14:58:18 dist-test-slave-15hz krb5kdc[11160](info): commencing operation
krb5kdc: starting...
W20260925 14:58:19.451041 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:19.561890 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42765
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:34423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:19.640275 11173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:19.640491 11173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:19.640522 11173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:19.641687 11173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:19.641726 11173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:19.641745 11173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:19.641762 11173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:19.641777 11173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:19.643285 11173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42765
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:19.643543 11173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:19.643761 11173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:19.648473 11173 server_base.cc:1034] running on GCE node
W20260925 14:58:19.648387 11181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.648640 11178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:19.648528 11179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:19.648902 11173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:19.649190 11173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:19.650326 11173 hybrid_clock.cc:648] HybridClock initialized: now 1790348299650307 us; error 35 us; skew 500 ppm
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:19.658370 11173 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:19.658926 11173 webserver.cc:460] Webserver started at http://127.8.13.254:43815/ using document root <none> and password file <none>
I20260925 14:58:19.659168 11173 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:19.659219 11173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:19.659353 11173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:19.660497 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "b556e9d6727c425793573f2858178a63"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-15hz"
server_key: "6d3e4394319b517652c6edc80a890dee"
server_key_iv: "c05fffb6f2651ed1c4dbfc866cbe73f2"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.660831 11173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "b556e9d6727c425793573f2858178a63"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-15hz"
server_key: "6d3e4394319b517652c6edc80a890dee"
server_key_iv: "c05fffb6f2651ed1c4dbfc866cbe73f2"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.662159 11173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:19.662894 11188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:19.663100 11173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:19.663190 11173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "b556e9d6727c425793573f2858178a63"
format_stamp: "Formatted at 2026-09-25 14:58:19 on dist-test-slave-15hz"
server_key: "6d3e4394319b517652c6edc80a890dee"
server_key_iv: "c05fffb6f2651ed1c4dbfc866cbe73f2"
server_key_version: "encryptionkey@0"
I20260925 14:58:19.663262 11173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:19.672461 11173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:19.674341 11173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:19.674476 11173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:19.678187 11173 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42765
I20260925 14:58:19.678226 11240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42765 every 8 connection(s)
I20260925 14:58:19.678560 11173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:19.679366 11241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:19.681644 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63: Bootstrap starting.
I20260925 14:58:19.682224 11241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:19.682488 11241 log.cc:824] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:19.683146 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63: No bootstrap required, opened a new log
I20260925 14:58:19.684772 11241 raft_consensus.cc:348] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } }
I20260925 14:58:19.684917 11241 raft_consensus.cc:374] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:19.684939 11241 raft_consensus.cc:729] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b556e9d6727c425793573f2858178a63, State: Initialized, Role: FOLLOWER
I20260925 14:58:19.685086 11241 consensus_queue.cc:260] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } }
I20260925 14:58:19.685151 11241 raft_consensus.cc:388] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:19.685173 11241 raft_consensus.cc:482] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:19.685228 11241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:19.686007 11241 raft_consensus.cc:504] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } }
I20260925 14:58:19.686101 11241 leader_election.cc:302] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b556e9d6727c425793573f2858178a63; no voters:
I20260925 14:58:19.686275 11241 leader_election.cc:288] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:19.686321 11244 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:19.686429 11244 raft_consensus.cc:686] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [term 1 LEADER]: Becoming Leader. State: Replica: b556e9d6727c425793573f2858178a63, State: Running, Role: LEADER
I20260925 14:58:19.686439 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11173
I20260925 14:58:19.686527 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:19.686511 11244 consensus_queue.cc:237] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } }
I20260925 14:58:19.686550 11241 sys_catalog.cc:564] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:19.686707 8247 external_mini_cluster.cc:1428] Setting key 471469be1bb17b5c78ecc7e220a327c4
I20260925 14:58:19.686860 11245 sys_catalog.cc:455] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b556e9d6727c425793573f2858178a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } } }
I20260925 14:58:19.687140 11245 sys_catalog.cc:458] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:19.687073 11246 sys_catalog.cc:455] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b556e9d6727c425793573f2858178a63. Latest consensus state: current_term: 1 leader_uuid: "b556e9d6727c425793573f2858178a63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b556e9d6727c425793573f2858178a63" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42765 } } }
I20260925 14:58:19.687397 11246 sys_catalog.cc:458] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:19.687644 11254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:19.688174 11254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:19.689744 11254 catalog_manager.cc:1347] Generated new cluster ID: 84b25eafbf034cfb8fb9f6e9b1b13b83
I20260925 14:58:19.689793 11254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348299, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:19.696405 11254 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:19.696952 11254 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:19.698680 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.687418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53138 (local address 127.8.13.254:42765)
0925 14:58:19.687733 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:19.687742 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:19.687779 (+ 37us) server_negotiation.cc:407] Connection header received
0925 14:58:19.688008 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:19.688016 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:19.688089 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:19.688250 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:19.688988 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.690178 (+ 1190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.691233 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.691873 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.691970 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.694792 (+ 2822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:19.694797 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:19.694797 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:19.694813 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:19.697023 (+ 2210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.697430 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.697433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.697434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.697558 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:19.697744 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:19.697745 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:19.697745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:19.698042 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:19.698208 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:19.698279 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 14:58:19.698444 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":67,"threads_started":1}
I20260925 14:58:19.699595 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:19.704834 11254 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b556e9d6727c425793573f2858178a63: Generated new TSK 0
I20260925 14:58:19.705029 11254 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:19.715487 11256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:19.711830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53148 (local address 127.8.13.254:42765)
0925 14:58:19.711879 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:19.711881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:19.711965 (+ 84us) server_negotiation.cc:407] Connection header received
0925 14:58:19.712035 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:19.712036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:19.712047 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:19.712095 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:19.712542 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.713064 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.714147 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:19.714357 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:19.714404 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:19.715058 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:19.715059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:19.715060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:19.715253 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:19.715271 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:19.715277 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:19.715319 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:19.715387 (+ 68us) server_negotiation.cc:299] Negotiation successful
0925 14:58:19.715416 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:58:19.717998 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11173
2026-09-25T14:58:19Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1218 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11274](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11274](info): set up 2 sockets
Sep 25 14:58:19 dist-test-slave-15hz krb5kdc[11274](info): commencing operation
krb5kdc: starting...
W20260925 14:58:20.667070 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:20 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:20.778347 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:38497
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:38497
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:20.857842 11287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:20.858062 11287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:20.858091 11287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:20.859283 11287 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:20.859329 11287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:20.859347 11287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:20.859364 11287 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:20.859378 11287 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:20.860891 11287 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:38497
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:38497
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:20.861147 11287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:20.861364 11287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:20.865839 11292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.865916 11293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:20.865869 11295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:20.866133 11287 server_base.cc:1034] running on GCE node
I20260925 14:58:20.866369 11287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:20.866632 11287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:20.867764 11287 hybrid_clock.cc:648] HybridClock initialized: now 1790348300867751 us; error 28 us; skew 500 ppm
Sep 25 14:58:20 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:20.870023 11287 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:20.870500 11287 webserver.cc:460] Webserver started at http://127.8.13.254:40455/ using document root <none> and password file <none>
I20260925 14:58:20.870697 11287 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:20.870745 11287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:20.870870 11287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:20.871961 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "938d68f8b25d4c31a6d0c4083f120842"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-15hz"
server_key: "5e893490e6dfa4aca70bb705296f54b5"
server_key_iv: "472f96d9b460c981de8dc3dd7190eaba"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.872313 11287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "938d68f8b25d4c31a6d0c4083f120842"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-15hz"
server_key: "5e893490e6dfa4aca70bb705296f54b5"
server_key_iv: "472f96d9b460c981de8dc3dd7190eaba"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.873510 11287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:20.874109 11302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:20.874253 11287 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:20.874328 11287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "938d68f8b25d4c31a6d0c4083f120842"
format_stamp: "Formatted at 2026-09-25 14:58:20 on dist-test-slave-15hz"
server_key: "5e893490e6dfa4aca70bb705296f54b5"
server_key_iv: "472f96d9b460c981de8dc3dd7190eaba"
server_key_version: "encryptionkey@0"
I20260925 14:58:20.874390 11287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:20.902776 11287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:20.904625 11287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:20.904769 11287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:20.908478 11287 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:38497
I20260925 14:58:20.908517 11354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:38497 every 8 connection(s)
I20260925 14:58:20.908842 11287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:20.909802 11355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:20.912130 11355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842: Bootstrap starting.
I20260925 14:58:20.912703 11355 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:20.912956 11355 log.cc:824] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:20.912992 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11287
I20260925 14:58:20.913059 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:20.913215 8247 external_mini_cluster.cc:1428] Setting key 74a31ebaccf58e868d219d2f03457e9f
I20260925 14:58:20.913643 11355 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842: No bootstrap required, opened a new log
I20260925 14:58:20.915370 11355 raft_consensus.cc:348] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } }
I20260925 14:58:20.915516 11355 raft_consensus.cc:374] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:20.915575 11355 raft_consensus.cc:729] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 938d68f8b25d4c31a6d0c4083f120842, State: Initialized, Role: FOLLOWER
I20260925 14:58:20.915706 11355 consensus_queue.cc:260] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } }
I20260925 14:58:20.915778 11355 raft_consensus.cc:388] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:20.915809 11355 raft_consensus.cc:482] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:20.915855 11355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:20.916648 11355 raft_consensus.cc:504] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } }
I20260925 14:58:20.916749 11355 leader_election.cc:302] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 938d68f8b25d4c31a6d0c4083f120842; no voters:
I20260925 14:58:20.916904 11355 leader_election.cc:288] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:20.916967 11360 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:20.917079 11360 raft_consensus.cc:686] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [term 1 LEADER]: Becoming Leader. State: Replica: 938d68f8b25d4c31a6d0c4083f120842, State: Running, Role: LEADER
I20260925 14:58:20.917167 11355 sys_catalog.cc:564] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:20.917156 11360 consensus_queue.cc:237] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } }
I20260925 14:58:20.917474 11360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 938d68f8b25d4c31a6d0c4083f120842. Latest consensus state: current_term: 1 leader_uuid: "938d68f8b25d4c31a6d0c4083f120842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } } }
I20260925 14:58:20.917537 11360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:20.917496 11361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "938d68f8b25d4c31a6d0c4083f120842" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "938d68f8b25d4c31a6d0c4083f120842" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38497 } } }
I20260925 14:58:20.917579 11361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:20.918341 11365 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:20.918864 11365 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 14:58:20 dist-test-slave-15hz krb5kdc[11274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:20.921782 11365 catalog_manager.cc:1347] Generated new cluster ID: 2552ed1950e54f14aca09af9e1fc6f34
I20260925 14:58:20.921830 11365 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:20.925580 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:20.913744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.8.13.254:38497)
0925 14:58:20.913902 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:20.913910 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:20.913955 (+ 45us) server_negotiation.cc:407] Connection header received
0925 14:58:20.915370 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:20.915379 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:20.915441 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:20.915628 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:20.916301 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.917436 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.918520 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:20.918998 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:20.919097 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:20.921710 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:20.921714 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:20.921714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:20.921728 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:20.923637 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.924025 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.924028 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.924029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.924150 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:20.924346 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:20.924347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:20.924348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:20.924634 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:20.925166 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:20.925242 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:58:20.925380 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":67,"threads_started":1}
I20260925 14:58:20.934000 11365 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:20.934870 11365 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:20.939723 11365 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 938d68f8b25d4c31a6d0c4083f120842: Generated new TSK 0
I20260925 14:58:20.939917 11365 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:20.973888 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:21.070662 11381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:21.070878 11381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:21.070902 11381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:21.072125 11381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:21.072166 11381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:21.072190 11381 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:21.073640 11381 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:21.073836 11381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:21.074021 11381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:21.079053 11381 server_base.cc:1034] running on GCE node
W20260925 14:58:21.079658 11387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.079938 11386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.080157 11389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.080372 11381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:21.080595 11381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:21.081730 11381 hybrid_clock.cc:648] HybridClock initialized: now 1790348301081719 us; error 28 us; skew 500 ppm
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:21.084173 11381 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:21.084616 11381 webserver.cc:460] Webserver started at http://127.8.13.193:46089/ using document root <none> and password file <none>
I20260925 14:58:21.084837 11381 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:21.084889 11381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:21.085019 11381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:21.086105 11381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "88794f8b5c434ae3b5e3d61fe0ffec76"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "abdbaf857a1bdaa4d66992296af2e2b5"
server_key_iv: "c4360564626769df34cfd3429358d30c"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.086462 11381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "88794f8b5c434ae3b5e3d61fe0ffec76"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "abdbaf857a1bdaa4d66992296af2e2b5"
server_key_iv: "c4360564626769df34cfd3429358d30c"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.087783 11381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:21.088465 11396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.088644 11381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:21.088734 11381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "88794f8b5c434ae3b5e3d61fe0ffec76"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "abdbaf857a1bdaa4d66992296af2e2b5"
server_key_iv: "c4360564626769df34cfd3429358d30c"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.088798 11381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:21.098851 11381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:21.100807 11381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:21.100945 11381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:21.101151 11381 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:21.101444 11381 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:21.101477 11381 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.101505 11381 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:21.101528 11381 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.107178 11381 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:40937
I20260925 14:58:21.107229 11509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:40937 every 8 connection(s)
I20260925 14:58:21.107599 11381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:21.109695 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11381
I20260925 14:58:21.109779 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:21.109953 8247 external_mini_cluster.cc:1428] Setting key 81f185af5031f08efc43b80340d8c89f
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:21.117821 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.108333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:42383 (local address 127.8.13.254:38497)
0925 14:58:21.108393 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.108395 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.108568 (+ 173us) server_negotiation.cc:407] Connection header received
0925 14:58:21.109052 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.109053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.109067 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.109130 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.110148 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.110833 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.111929 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.112155 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.112223 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.114656 (+ 2433us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.114657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.114658 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.114668 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.116094 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.116600 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.116600 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.116601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.116654 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.116970 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.116971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.116972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.117164 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.117238 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.117613 (+ 375us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.117746 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:21.118008 11512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.108299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:38497 (local address 127.8.13.193:42383)
0925 14:58:21.108445 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.108488 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.108974 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.109191 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.109196 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.109791 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:21.110043 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.110050 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.110928 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.110929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.111845 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.111849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.112233 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.112234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.112361 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.112884 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:21.112893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:21.114548 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:21.116182 (+ 1634us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.116191 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.116194 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.116525 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.116708 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.116709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.116709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.116855 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.117252 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:21.117260 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:21.117449 (+ 189us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.117578 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.117766 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 14:58:21.118676 11510 heartbeater.cc:347] Connected to a master server at 127.8.13.254:38497
I20260925 14:58:21.118769 11510 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:21.118947 11510 heartbeater.cc:511] Master 127.8.13.254:38497 requested a full tablet report, sending...
I20260925 14:58:21.119498 11319 ts_manager.cc:194] Registered new tserver with Master: 88794f8b5c434ae3b5e3d61fe0ffec76 (127.8.13.193:40937)
I20260925 14:58:21.120133 11319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:42383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:21.134730 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:21.213371 11515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:21.213587 11515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:21.213611 11515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:21.214761 11515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:21.214798 11515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:21.214824 11515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:21.216269 11515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:21.216470 11515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:21.216663 11515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:21.221519 11523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.221601 11521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.221510 11520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.221837 11515 server_base.cc:1034] running on GCE node
I20260925 14:58:21.222048 11515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:21.222275 11515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:21.223412 11515 hybrid_clock.cc:648] HybridClock initialized: now 1790348301223398 us; error 29 us; skew 500 ppm
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:21.225845 11515 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:21.226271 11515 webserver.cc:460] Webserver started at http://127.8.13.194:34917/ using document root <none> and password file <none>
I20260925 14:58:21.226470 11515 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:21.226514 11515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:21.226621 11515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:21.227731 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "9d844c8e1fde4c43baf8d8d80d12b841"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "c4e2d30025345284bf5751c8f5830bab"
server_key_iv: "27e47251a14894f63ed4ee9984e56fe3"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.228075 11515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "9d844c8e1fde4c43baf8d8d80d12b841"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "c4e2d30025345284bf5751c8f5830bab"
server_key_iv: "27e47251a14894f63ed4ee9984e56fe3"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.229398 11515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:21.230091 11530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.230268 11515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.230356 11515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "9d844c8e1fde4c43baf8d8d80d12b841"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "c4e2d30025345284bf5751c8f5830bab"
server_key_iv: "27e47251a14894f63ed4ee9984e56fe3"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.230417 11515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:21.245468 11515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:21.247659 11515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:21.247777 11515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:21.247982 11515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:21.248246 11515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:21.248274 11515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.248306 11515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:21.248325 11515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.253769 11515 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:46569
I20260925 14:58:21.253818 11643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:46569 every 8 connection(s)
I20260925 14:58:21.254115 11515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:21.259481 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11515
I20260925 14:58:21.259593 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:21.259776 8247 external_mini_cluster.cc:1428] Setting key eec8f92a0f1e78ae957d7be2dfa92181
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:21.262878 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.254783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:37181 (local address 127.8.13.254:38497)
0925 14:58:21.254852 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.254855 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.255033 (+ 178us) server_negotiation.cc:407] Connection header received
0925 14:58:21.255503 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.255504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.255514 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.255600 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.256486 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.256986 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.257939 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.258116 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.258158 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.260276 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.260278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.260278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.260284 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.261542 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.261897 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.261898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.261899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.261953 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.262204 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.262205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.262206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.262360 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.262427 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.262696 (+ 269us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.262790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:21.262974 11646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.254764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:38497 (local address 127.8.13.194:37181)
0925 14:58:21.254920 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.254956 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.255431 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.255672 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.255676 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.256222 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:21.256419 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.256424 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.257058 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.257059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.257881 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.257883 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.258169 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.258170 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.258264 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.258751 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:21.258759 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:21.260182 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:21.261615 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.261623 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.261625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.261834 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.262003 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.262004 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.262005 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.262124 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.262429 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:21.262434 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:21.262553 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.262634 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.262790 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260925 14:58:21.263607 11644 heartbeater.cc:347] Connected to a master server at 127.8.13.254:38497
I20260925 14:58:21.263712 11644 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:21.263885 11644 heartbeater.cc:511] Master 127.8.13.254:38497 requested a full tablet report, sending...
I20260925 14:58:21.264263 11319 ts_manager.cc:194] Registered new tserver with Master: 9d844c8e1fde4c43baf8d8d80d12b841 (127.8.13.194:46569)
I20260925 14:58:21.264629 11319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:37181
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:21.283481 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:34833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:21.363066 11649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:21.363276 11649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:21.363298 11649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:21.364427 11649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:21.364463 11649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:21.364486 11649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:21.365886 11649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:38497
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:21.366072 11649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:21.366259 11649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:21.370909 11657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.370937 11655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:21.370932 11654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:21.370963 11649 server_base.cc:1034] running on GCE node
I20260925 14:58:21.371344 11649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:21.371563 11649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:21.372709 11649 hybrid_clock.cc:648] HybridClock initialized: now 1790348301372692 us; error 32 us; skew 500 ppm
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:21.374981 11649 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:21.375413 11649 webserver.cc:460] Webserver started at http://127.8.13.195:37763/ using document root <none> and password file <none>
I20260925 14:58:21.375661 11649 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:21.375725 11649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:21.375838 11649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:21.376848 11649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "3610c293f46d4996a1da03d30f391a32"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "5484c52bebeb677ebf846bf286eaefa9"
server_key_iv: "93350f785ce1e5e4fe7cbb1c17a853a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.377149 11649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "3610c293f46d4996a1da03d30f391a32"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "5484c52bebeb677ebf846bf286eaefa9"
server_key_iv: "93350f785ce1e5e4fe7cbb1c17a853a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.378365 11649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:21.379048 11664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.379215 11649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.379287 11649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "3610c293f46d4996a1da03d30f391a32"
format_stamp: "Formatted at 2026-09-25 14:58:21 on dist-test-slave-15hz"
server_key: "5484c52bebeb677ebf846bf286eaefa9"
server_key_iv: "93350f785ce1e5e4fe7cbb1c17a853a9"
server_key_version: "encryptionkey@0"
I20260925 14:58:21.379354 11649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:21.413892 11649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:21.415791 11649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:21.415913 11649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:21.416144 11649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:21.416464 11649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:21.416498 11649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.416527 11649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:21.416548 11649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:21.422475 11649 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:35181
I20260925 14:58:21.422516 11777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:35181 every 8 connection(s)
I20260925 14:58:21.422827 11649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:21.428474 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11649
I20260925 14:58:21.428586 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:21.428779 8247 external_mini_cluster.cc:1428] Setting key 7eaeef01c1c14d5495ae41d8acc0c583
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:21.431691 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.423552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:34109 (local address 127.8.13.254:38497)
0925 14:58:21.423597 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.423599 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.423795 (+ 196us) server_negotiation.cc:407] Connection header received
0925 14:58:21.424301 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.424302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.424314 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.424371 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.425318 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.425826 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.426799 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.427001 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.427057 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.429249 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.429250 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.429251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.429259 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.430409 (+ 1150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.430781 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.430782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.430783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.430837 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.431050 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.431051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.431052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.431164 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.431221 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.431489 (+ 268us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.431601 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:21.431875 11780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.423502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:38497 (local address 127.8.13.195:34109)
0925 14:58:21.423675 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:21.423715 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:21.424209 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:21.424446 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:21.424450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:21.425022 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:21.425227 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.425232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.425912 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.425913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.426721 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:21.426723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.427071 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.427072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:21.427185 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.427690 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:21.427699 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:21.429153 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:21.430479 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.430487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.430489 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.430716 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.430891 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:21.430891 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:21.430891 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:21.430996 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:21.431229 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:21.431235 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:21.431373 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 14:58:21.431463 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:58:21.431655 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":107,"threads_started":1}
I20260925 14:58:21.432301 11778 heartbeater.cc:347] Connected to a master server at 127.8.13.254:38497
I20260925 14:58:21.432395 11778 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:21.432550 11778 heartbeater.cc:511] Master 127.8.13.254:38497 requested a full tablet report, sending...
I20260925 14:58:21.432858 11319 ts_manager.cc:194] Registered new tserver with Master: 3610c293f46d4996a1da03d30f391a32 (127.8.13.195:35181)
I20260925 14:58:21.433267 11319 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:34109
I20260925 14:58:21.440507 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11274](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348301, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:21.461203 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:21.454987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56400 (local address 127.8.13.254:38497)
0925 14:58:21.455055 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:21.455057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:21.455144 (+ 87us) server_negotiation.cc:407] Connection header received
0925 14:58:21.455208 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:21.455209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:21.455217 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:21.455264 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:21.455806 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.456299 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.456961 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:21.457133 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:21.457172 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:21.459225 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:21.459226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:21.459227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:21.459234 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:21.460361 (+ 1127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.460630 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.460630 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.460631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.460674 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:21.460792 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:21.460792 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:21.460792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:21.460927 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:21.460999 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:21.461063 (+ 64us) server_negotiation.cc:299] Negotiation successful
0925 14:58:21.461121 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260925 14:58:21.461453 11319 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56400
I20260925 14:58:21.461894 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11381
I20260925 14:58:21.465577 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11515
I20260925 14:58:21.469106 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11649
I20260925 14:58:21.472654 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11287
2026-09-25T14:58:21Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1755 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11790](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11790](info): set up 2 sockets
Sep 25 14:58:21 dist-test-slave-15hz krb5kdc[11790](info): commencing operation
krb5kdc: starting...
W20260925 14:58:22.415201 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:22.528600 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43313
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:22.607810 11803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:22.608036 11803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:22.608067 11803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:22.609264 11803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:22.609305 11803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:22.609323 11803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:22.609340 11803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:22.609354 11803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:22.610937 11803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:43313
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:22.611194 11803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:22.611405 11803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:22.615953 11808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.616022 11809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.615947 11811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.616235 11803 server_base.cc:1034] running on GCE node
I20260925 14:58:22.616457 11803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:22.616739 11803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:22.617903 11803 hybrid_clock.cc:648] HybridClock initialized: now 1790348302617888 us; error 28 us; skew 500 ppm
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:22.620354 11803 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:22.620771 11803 webserver.cc:460] Webserver started at http://127.8.13.254:43889/ using document root <none> and password file <none>
I20260925 14:58:22.620967 11803 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:22.621014 11803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:22.621148 11803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:22.622202 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "6fd45fba89eb4910b736448042b4042b"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "2d4d2d40a522f37c7f262124bec2dc68"
server_key_iv: "e2d49116c0f1acebb38d604668aa9fec"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.622531 11803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "6fd45fba89eb4910b736448042b4042b"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "2d4d2d40a522f37c7f262124bec2dc68"
server_key_iv: "e2d49116c0f1acebb38d604668aa9fec"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.623910 11803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:22.624609 11818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.624780 11803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:22.624862 11803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "6fd45fba89eb4910b736448042b4042b"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "2d4d2d40a522f37c7f262124bec2dc68"
server_key_iv: "e2d49116c0f1acebb38d604668aa9fec"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.624926 11803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:22.632505 11803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:22.634428 11803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:22.634574 11803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:22.638449 11803 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:43313
I20260925 14:58:22.638490 11870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:43313 every 8 connection(s)
I20260925 14:58:22.638949 11803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:22.639657 11871 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:22.641940 11871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b: Bootstrap starting.
I20260925 14:58:22.642520 11871 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:22.642788 11871 log.cc:824] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:22.643101 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11803
I20260925 14:58:22.643199 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:22.643374 8247 external_mini_cluster.cc:1428] Setting key 0767076a8f08d956550c0b0e94e8f642
I20260925 14:58:22.643477 11871 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b: No bootstrap required, opened a new log
I20260925 14:58:22.646131 11871 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } }
I20260925 14:58:22.646306 11871 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:22.646337 11871 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd45fba89eb4910b736448042b4042b, State: Initialized, Role: FOLLOWER
I20260925 14:58:22.646450 11871 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } }
I20260925 14:58:22.646519 11871 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:22.646553 11871 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:22.646595 11871 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:22.647240 11871 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } }
I20260925 14:58:22.647332 11871 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd45fba89eb4910b736448042b4042b; no voters:
I20260925 14:58:22.647480 11871 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:22.647506 11876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:22.647646 11876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [term 1 LEADER]: Becoming Leader. State: Replica: 6fd45fba89eb4910b736448042b4042b, State: Running, Role: LEADER
I20260925 14:58:22.647725 11876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } }
I20260925 14:58:22.647758 11871 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:22.648025 11876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd45fba89eb4910b736448042b4042b. Latest consensus state: current_term: 1 leader_uuid: "6fd45fba89eb4910b736448042b4042b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } } }
I20260925 14:58:22.648090 11876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:22.648090 11878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd45fba89eb4910b736448042b4042b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd45fba89eb4910b736448042b4042b" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43313 } } }
I20260925 14:58:22.648197 11878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:22.648355 11881 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:22.648886 11881 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:22.650471 11881 catalog_manager.cc:1347] Generated new cluster ID: 0923d091210444f5899c79d4dd0d482e
I20260925 14:58:22.650524 11881 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:22.654112 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.643922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60928 (local address 127.8.13.254:43313)
0925 14:58:22.644100 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.644111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.644150 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:22.644793 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.644803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.644868 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.645012 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.645594 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.646937 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.647856 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.648498 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.648581 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.651167 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.651170 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.651171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.651183 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.652750 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.653068 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.653071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.653072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.653159 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.653326 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.653327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.653328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.653572 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.653721 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.653812 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.653943 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260925 14:58:22.663929 11881 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:22.664593 11881 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:22.669378 11881 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd45fba89eb4910b736448042b4042b: Generated new TSK 0
I20260925 14:58:22.669564 11881 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:22.696382 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:22.801913 11897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:22.802131 11897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:22.802155 11897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:22.803352 11897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:22.803398 11897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:22.803426 11897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:22.805058 11897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:22.805300 11897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:22.805526 11897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:22.810446 11902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.810621 11897 server_base.cc:1034] running on GCE node
W20260925 14:58:22.810446 11905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.810446 11903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.810927 11897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:22.811151 11897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:22.812279 11897 hybrid_clock.cc:648] HybridClock initialized: now 1790348302812263 us; error 30 us; skew 500 ppm
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:22.814620 11897 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:22.815081 11897 webserver.cc:460] Webserver started at http://127.8.13.193:43893/ using document root <none> and password file <none>
I20260925 14:58:22.815289 11897 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:22.815335 11897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:22.815454 11897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:22.816526 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "ebee9a3701da4083a6aceb9e5b8a85c7"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "8deab24f063822922c774a9727f6112b"
server_key_iv: "4cd6f7180a982846163374fa90690b3e"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.816859 11897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "ebee9a3701da4083a6aceb9e5b8a85c7"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "8deab24f063822922c774a9727f6112b"
server_key_iv: "4cd6f7180a982846163374fa90690b3e"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.818094 11897 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:22.818863 11912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.819048 11897 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:22.819154 11897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ebee9a3701da4083a6aceb9e5b8a85c7"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "8deab24f063822922c774a9727f6112b"
server_key_iv: "4cd6f7180a982846163374fa90690b3e"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.819214 11897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:22.826344 11897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:22.828505 11897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:22.828639 11897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:22.828837 11897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:22.829134 11897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:22.829167 11897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.829195 11897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:22.829219 11897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.834744 11897 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:37661
I20260925 14:58:22.834791 12025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:37661 every 8 connection(s)
I20260925 14:58:22.835114 11897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:22.841969 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 11897
I20260925 14:58:22.842073 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:22.842255 8247 external_mini_cluster.cc:1428] Setting key a7c098652c1208b8065d60bd0ddc3b01
I20260925 14:58:22.844129 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.835831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:45721 (local address 127.8.13.254:43313)
0925 14:58:22.835907 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.835909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.836114 (+ 205us) server_negotiation.cc:407] Connection header received
0925 14:58:22.837012 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.837013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.837023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.837074 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.837753 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.838290 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.839238 (+ 948us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.839426 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.839468 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.841473 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:22.841474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:22.841474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:22.841480 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:22.842814 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.843205 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.843206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.843207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.843270 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:22.843494 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:22.843495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:22.843495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:22.843652 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:22.843712 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:22.843946 (+ 234us) server_negotiation.cc:299] Negotiation successful
0925 14:58:22.844032 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260925 14:58:22.844493 12028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.835798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43313 (local address 127.8.13.193:45721)
0925 14:58:22.835981 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.836025 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.836930 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.837129 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.837133 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.837448 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:22.837680 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.837684 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.838360 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.838360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.839178 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.839180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.839478 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.839478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.839624 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.840100 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:22.840109 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:22.841391 (+ 1282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:22.842888 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.842896 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.842899 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.843139 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.843328 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:22.843329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:22.843329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:22.843443 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:22.843715 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:22.843720 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:22.843839 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:58:22.843917 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:58:22.844041 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
I20260925 14:58:22.844990 12026 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43313
I20260925 14:58:22.845098 12026 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:22.845287 12026 heartbeater.cc:511] Master 127.8.13.254:43313 requested a full tablet report, sending...
I20260925 14:58:22.845758 11835 ts_manager.cc:194] Registered new tserver with Master: ebee9a3701da4083a6aceb9e5b8a85c7 (127.8.13.193:37661)
I20260925 14:58:22.846475 11835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:45721
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:22.867033 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:22.948486 12031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:22.948696 12031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:22.948727 12031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:22.949925 12031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:22.949965 12031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:22.950001 12031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:22.951471 12031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:22.951735 12031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:22.951934 12031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:22.956516 12037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.956487 12039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:22.956480 12036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:22.956964 12031 server_base.cc:1034] running on GCE node
I20260925 14:58:22.957172 12031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:22.957360 12031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:22.958485 12031 hybrid_clock.cc:648] HybridClock initialized: now 1790348302958474 us; error 30 us; skew 500 ppm
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:22.960958 12031 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:22.961473 12031 webserver.cc:460] Webserver started at http://127.8.13.194:38753/ using document root <none> and password file <none>
I20260925 14:58:22.961670 12031 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:22.961711 12031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:22.961835 12031 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:22.962839 12031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "ea6c961cc5ee4a89a51d3092188c2832"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "662ea1f1682530115eb0d0551d46c478"
server_key_iv: "dfcc3f905190022ec6ccddcdad37d107"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.963162 12031 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "ea6c961cc5ee4a89a51d3092188c2832"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "662ea1f1682530115eb0d0551d46c478"
server_key_iv: "dfcc3f905190022ec6ccddcdad37d107"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.964401 12031 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:22.965050 12046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.965201 12031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:22.965274 12031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "ea6c961cc5ee4a89a51d3092188c2832"
format_stamp: "Formatted at 2026-09-25 14:58:22 on dist-test-slave-15hz"
server_key: "662ea1f1682530115eb0d0551d46c478"
server_key_iv: "dfcc3f905190022ec6ccddcdad37d107"
server_key_version: "encryptionkey@0"
I20260925 14:58:22.965337 12031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:22.985185 12031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:22.987339 12031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:22.987457 12031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:22.987713 12031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:22.988003 12031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:22.988034 12031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.988077 12031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:22.988101 12031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:22.993580 12031 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:38949
I20260925 14:58:22.993631 12159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:38949 every 8 connection(s)
I20260925 14:58:22.993928 12031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:22 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348302, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:23.001951 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12031
I20260925 14:58:23.002092 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:23.002270 8247 external_mini_cluster.cc:1428] Setting key 4c048bdb420f1a3b749afa7f376cee52
I20260925 14:58:23.002607 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.994682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:35621 (local address 127.8.13.254:43313)
0925 14:58:22.994735 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:22.994737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:22.994928 (+ 191us) server_negotiation.cc:407] Connection header received
0925 14:58:22.995858 (+ 930us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:22.995859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:22.995870 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:22.995931 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:22.996509 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.997001 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.997999 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.998198 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.998251 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.000314 (+ 2063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:23.000316 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:23.000316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:23.000323 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:23.001380 (+ 1057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.001696 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.001696 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.001697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.001746 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.001938 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.001939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.001939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.002079 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:23.002146 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.002418 (+ 272us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.002502 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:58:23.002748 12162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:22.994657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43313 (local address 127.8.13.194:35621)
0925 14:58:22.994808 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:22.994851 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:22.995756 (+ 905us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:22.995989 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:22.995992 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:22.996251 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:22.996445 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.996449 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.997093 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.997093 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.997912 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:22.997915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:22.998264 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:22.998265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:22.998366 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:22.998864 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:22.998872 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:23.000227 (+ 1355us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:23.001441 (+ 1214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:23.001446 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:23.001448 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:23.001644 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:23.001802 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:23.001803 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:23.001803 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:23.001888 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:23.002136 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:23.002143 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:23.002279 (+ 136us) client_negotiation.cc:769] Sending connection context
0925 14:58:23.002384 (+ 105us) client_negotiation.cc:240] Negotiation successful
0925 14:58:23.002535 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260925 14:58:23.003131 12160 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43313
I20260925 14:58:23.003216 12160 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:23.003382 12160 heartbeater.cc:511] Master 127.8.13.254:43313 requested a full tablet report, sending...
I20260925 14:58:23.003775 11835 ts_manager.cc:194] Registered new tserver with Master: ea6c961cc5ee4a89a51d3092188c2832 (127.8.13.194:38949)
I20260925 14:58:23.004215 11835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:35621
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:23.024098 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:23.103873 12165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:23.104090 12165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:23.104111 12165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:23.105247 12165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:23.105284 12165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:23.105309 12165 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:23.106765 12165 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12165
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:23.106968 12165 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:23.107153 12165 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:23.111845 12171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:23.111979 12165 server_base.cc:1034] running on GCE node
W20260925 14:58:23.111886 12173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:23.111894 12170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:23.112265 12165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:23.112507 12165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:23.113647 12165 hybrid_clock.cc:648] HybridClock initialized: now 1790348303113625 us; error 35 us; skew 500 ppm
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:23.116082 12165 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:23.116520 12165 webserver.cc:460] Webserver started at http://127.8.13.195:35483/ using document root <none> and password file <none>
I20260925 14:58:23.116737 12165 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:23.116786 12165 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:23.116909 12165 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:23.117944 12165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "9d900377dd474707a44ac06006d77e46"
format_stamp: "Formatted at 2026-09-25 14:58:23 on dist-test-slave-15hz"
server_key: "fb6cfbf93c420f23c581e1cbde716e3e"
server_key_iv: "545203e744bdbd8e5931a548cd8ca17f"
server_key_version: "encryptionkey@0"
I20260925 14:58:23.118287 12165 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "9d900377dd474707a44ac06006d77e46"
format_stamp: "Formatted at 2026-09-25 14:58:23 on dist-test-slave-15hz"
server_key: "fb6cfbf93c420f23c581e1cbde716e3e"
server_key_iv: "545203e744bdbd8e5931a548cd8ca17f"
server_key_version: "encryptionkey@0"
I20260925 14:58:23.119482 12165 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:23.120193 12180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:23.120337 12165 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:23.120416 12165 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "9d900377dd474707a44ac06006d77e46"
format_stamp: "Formatted at 2026-09-25 14:58:23 on dist-test-slave-15hz"
server_key: "fb6cfbf93c420f23c581e1cbde716e3e"
server_key_iv: "545203e744bdbd8e5931a548cd8ca17f"
server_key_version: "encryptionkey@0"
I20260925 14:58:23.120476 12165 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:23.131278 12165 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:23.133287 12165 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:23.133414 12165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:23.133616 12165 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:23.133889 12165 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:23.133920 12165 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:23.133947 12165 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:23.133970 12165 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:23.139348 12165 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:46527
I20260925 14:58:23.139386 12293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:46527 every 8 connection(s)
I20260925 14:58:23.139761 12165 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:23.148373 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.140482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:37479 (local address 127.8.13.254:43313)
0925 14:58:23.140557 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:23.140559 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:23.140704 (+ 145us) server_negotiation.cc:407] Connection header received
0925 14:58:23.141609 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:23.141610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:23.141620 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:23.141665 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:23.142214 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.142703 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.143728 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.143907 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.143955 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.145859 (+ 1904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:23.145860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:23.145860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:23.145867 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:23.147124 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.147560 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.147560 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.147560 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.147615 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.147864 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.147865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.147865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.147966 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:23.148014 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.148242 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.148299 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 14:58:23.148617 12296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.140456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43313 (local address 127.8.13.195:37479)
0925 14:58:23.140597 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:23.140632 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:23.141521 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:23.141722 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:23.141725 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:23.141961 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:23.142154 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:23.142157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.142787 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.142788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:23.143661 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:23.143665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.143963 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.143964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:23.144059 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.144490 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:23.144497 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:23.145769 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:23.147206 (+ 1437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:23.147212 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:23.147214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:23.147470 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:23.147681 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:23.147681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:23.147682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:23.147813 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:23.148011 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:23.148017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:23.148147 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:58:23.148228 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:58:23.148390 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260925 14:58:23.148808 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12165
I20260925 14:58:23.148907 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:23.149109 8247 external_mini_cluster.cc:1428] Setting key d146d1d316682509efabcbe1f45b4414
I20260925 14:58:23.149132 12294 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43313
I20260925 14:58:23.149245 12294 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:23.149441 12294 heartbeater.cc:511] Master 127.8.13.254:43313 requested a full tablet report, sending...
I20260925 14:58:23.149972 11835 ts_manager.cc:194] Registered new tserver with Master: 9d900377dd474707a44ac06006d77e46 (127.8.13.195:46527)
I20260925 14:58:23.150350 11835 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:37479
I20260925 14:58:23.160811 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[11790](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:23.181625 12301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.174630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60090 (local address 127.8.13.193:37661)
0925 14:58:23.174777 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:23.174782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:23.174793 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:23.174818 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:23.174821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:23.174834 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:23.174897 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:23.175395 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.176167 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.176858 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.177176 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.177234 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.179254 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:23.179261 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:23.179261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:23.179268 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:23.180721 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.181011 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.181014 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.181014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.181072 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.181198 (+ 126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.181199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.181199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.181316 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:23.181413 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.181465 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.181520 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":74,"threads_started":1}
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[11790](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348303, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:23.188649 11875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:23.182776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.8.13.254:43313)
0925 14:58:23.182819 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:23.182820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:23.182828 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:23.182872 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:23.182872 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:23.182881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:23.182919 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:23.183314 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.183819 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.184435 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:23.184609 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:23.184650 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:23.186519 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:23.186520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:23.186520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:23.186526 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:23.187763 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.188061 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.188061 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.188062 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.188108 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:23.188249 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:23.188250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:23.188250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:23.188374 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:23.188441 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:23.188522 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:58:23.188572 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:58:23.188864 11835 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60944
I20260925 14:58:23.189218 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11897
I20260925 14:58:23.193248 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12031
I20260925 14:58:23.197052 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12165
I20260925 14:58:23.200532 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 11803
2026-09-25T14:58:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1727 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[12306](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[12306](info): set up 2 sockets
Sep 25 14:58:23 dist-test-slave-15hz krb5kdc[12306](info): commencing operation
krb5kdc: starting...
W20260925 14:58:24.147172 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:24.256163 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43997
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:43997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:24.335649 12319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:24.335873 12319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:24.335906 12319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:24.337113 12319 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:24.337154 12319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:24.337172 12319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:24.337189 12319 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:24.337203 12319 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:24.338733 12319 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:43997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:24.338982 12319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:24.339201 12319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:24.343808 12325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.343817 12327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.343845 12319 server_base.cc:1034] running on GCE node
W20260925 14:58:24.343817 12324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.344246 12319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:24.344522 12319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:24.345657 12319 hybrid_clock.cc:648] HybridClock initialized: now 1790348304345648 us; error 41 us; skew 500 ppm
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:24.347992 12319 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:24.348281 12319 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:24.348323 12319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:24.348412 12319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:24.349414 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "3d567740939a41849fc239a9a373d4af"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "01b57018ffaa02e8c1120728ae8a789f"
server_key_iv: "252d4c780e39b89ec61aa08eb2a0b757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.349704 12319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "3d567740939a41849fc239a9a373d4af"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "01b57018ffaa02e8c1120728ae8a789f"
server_key_iv: "252d4c780e39b89ec61aa08eb2a0b757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.350828 12319 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:24.351449 12333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.351594 12319 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.351668 12319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "3d567740939a41849fc239a9a373d4af"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "01b57018ffaa02e8c1120728ae8a789f"
server_key_iv: "252d4c780e39b89ec61aa08eb2a0b757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.351732 12319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:24.361549 12319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:24.363351 12319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:24.363473 12319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:24.367276 12319 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:43997
I20260925 14:58:24.367300 12385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:43997 every 8 connection(s)
I20260925 14:58:24.367609 12319 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:24.368482 12386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:24.370220 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12319
I20260925 14:58:24.370322 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:24.370491 8247 external_mini_cluster.cc:1428] Setting key 2b9f5a32d58028c2eb382d0284a052b5
I20260925 14:58:24.370985 12386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: Bootstrap starting.
I20260925 14:58:24.371578 12386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:24.371883 12386 log.cc:824] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:24.372572 12386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: No bootstrap required, opened a new log
I20260925 14:58:24.374562 12386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } }
I20260925 14:58:24.374708 12386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:24.374729 12386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d567740939a41849fc239a9a373d4af, State: Initialized, Role: FOLLOWER
I20260925 14:58:24.374840 12386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } }
I20260925 14:58:24.374905 12386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:24.374960 12386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:24.375010 12386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:24.375697 12386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } }
I20260925 14:58:24.375797 12386 leader_election.cc:302] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d567740939a41849fc239a9a373d4af; no voters:
I20260925 14:58:24.375946 12386 leader_election.cc:288] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:24.375981 12391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:24.376107 12391 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [term 1 LEADER]: Becoming Leader. State: Replica: 3d567740939a41849fc239a9a373d4af, State: Running, Role: LEADER
I20260925 14:58:24.376259 12386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:24.376238 12391 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } }
I20260925 14:58:24.376770 12393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d567740939a41849fc239a9a373d4af. Latest consensus state: current_term: 1 leader_uuid: "3d567740939a41849fc239a9a373d4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } } }
I20260925 14:58:24.376849 12393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [sys.catalog]: This master's current role is: LEADER
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:24.377071 12392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d567740939a41849fc239a9a373d4af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d567740939a41849fc239a9a373d4af" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43997 } } }
I20260925 14:58:24.377146 12392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:24.378049 12406 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:24.378103 12406 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:24.378161 12400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:24.378320 12400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:24.379801 12400 catalog_manager.cc:1347] Generated new cluster ID: fdd63d5665a143c0a376edc97da72982
I20260925 14:58:24.379843 12400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:24.380604 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.370975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60998 (local address 127.8.13.254:43997)
0925 14:58:24.371122 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.371131 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.371161 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:24.371627 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.371636 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.371699 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.371818 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:24.372475 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.373364 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.374408 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.374840 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.374942 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.377480 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:24.377484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:24.377485 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:24.377497 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:24.379073 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.379430 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.379433 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.379433 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.379511 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.379719 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.379720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.379720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.380012 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:24.380151 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.380289 (+ 138us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.380417 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260925 14:58:24.384312 12400 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:24.384788 12400 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:24.392032 12400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: Generated new TSK 0
I20260925 14:58:24.392232 12400 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:24.418643 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:24.498991 12412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:24.499225 12412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:24.499253 12412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:24.500496 12412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:24.500543 12412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:24.500572 12412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:24.502144 12412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:24.502370 12412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:24.502591 12412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:24.507404 12418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.507454 12420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.507406 12417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.507750 12412 server_base.cc:1034] running on GCE node
I20260925 14:58:24.507969 12412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:24.508183 12412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:24.509321 12412 hybrid_clock.cc:648] HybridClock initialized: now 1790348304509310 us; error 29 us; skew 500 ppm
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:24.511726 12412 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:24.512041 12412 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:24.512090 12412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:24.512210 12412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:24.513250 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "e565e23a6cda47c0b0385d9012ce6b3c"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "fbf1410c85b5c410d8088e00b5a01425"
server_key_iv: "e4c792f7b7536d56bb4bf1a90a5fa757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.513572 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "e565e23a6cda47c0b0385d9012ce6b3c"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "fbf1410c85b5c410d8088e00b5a01425"
server_key_iv: "e4c792f7b7536d56bb4bf1a90a5fa757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.514750 12412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:24.515368 12426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.515520 12412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.515637 12412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "e565e23a6cda47c0b0385d9012ce6b3c"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "fbf1410c85b5c410d8088e00b5a01425"
server_key_iv: "e4c792f7b7536d56bb4bf1a90a5fa757"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.515698 12412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:24.529101 12412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:24.530895 12412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:24.531037 12412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:24.531256 12412 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:24.531591 12412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:24.531646 12412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.531678 12412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:24.531702 12412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.537088 12412 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:39815
I20260925 14:58:24.537143 12539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:39815 every 8 connection(s)
I20260925 14:58:24.537379 12412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:24.543444 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12412
I20260925 14:58:24.543555 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:24.543730 8247 external_mini_cluster.cc:1428] Setting key d1db6b26af9fee3af222a42a9f8a3e0f
I20260925 14:58:24.546360 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.538054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:33221 (local address 127.8.13.254:43997)
0925 14:58:24.538106 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.538108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.538320 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:58:24.538814 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.538816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.538828 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.538879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:24.539840 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.540386 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.541478 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.541677 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.541728 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.543794 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:24.543795 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:24.543796 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:24.543804 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:24.545091 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.545468 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.545470 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.545470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.545527 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.545753 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.545754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.545754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.545868 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:24.545921 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.546183 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.546269 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:58:24.546541 12542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.538066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43997 (local address 127.8.13.193:33221)
0925 14:58:24.538206 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.538243 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.538721 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.538945 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.538949 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.539206 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:24.539749 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.539756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.540534 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.540535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.541400 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.541403 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.541745 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.541746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.541844 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.542322 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:24.542331 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:24.543701 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:24.545179 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.545186 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.545189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.545404 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.545581 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.545582 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.545582 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.545698 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.545921 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:24.545927 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:24.546067 (+ 140us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.546158 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.546321 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":66,"threads_started":1}
I20260925 14:58:24.546964 12540 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43997
I20260925 14:58:24.547070 12540 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:24.547230 12540 heartbeater.cc:511] Master 127.8.13.254:43997 requested a full tablet report, sending...
I20260925 14:58:24.547829 12350 ts_manager.cc:194] Registered new tserver with Master: e565e23a6cda47c0b0385d9012ce6b3c (127.8.13.193:39815)
I20260925 14:58:24.548738 12350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:33221
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:24.568485 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:24.647955 12545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:24.648165 12545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:24.648187 12545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:24.649346 12545 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:24.649389 12545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:24.649416 12545 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:24.650907 12545 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12545
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:24.651124 12545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:24.651345 12545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:24.655954 12550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.655954 12553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.656118 12545 server_base.cc:1034] running on GCE node
W20260925 14:58:24.655982 12551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.656407 12545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:24.656641 12545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:24.657805 12545 hybrid_clock.cc:648] HybridClock initialized: now 1790348304657791 us; error 28 us; skew 500 ppm
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:24.664855 12545 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:24.665210 12545 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:24.665259 12545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:24.665375 12545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:24.666421 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "3f7020ff58624728a75b29c84c18a8ba"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "5664fe0778fe11f49ee30927788ac3cc"
server_key_iv: "9b5e1b583fa4e196c79014fe11dd9c69"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.666767 12545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "3f7020ff58624728a75b29c84c18a8ba"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "5664fe0778fe11f49ee30927788ac3cc"
server_key_iv: "9b5e1b583fa4e196c79014fe11dd9c69"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.668020 12545 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:24.668754 12559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.668905 12545 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:24.668980 12545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "3f7020ff58624728a75b29c84c18a8ba"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "5664fe0778fe11f49ee30927788ac3cc"
server_key_iv: "9b5e1b583fa4e196c79014fe11dd9c69"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.669044 12545 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:24.687198 12545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:24.689481 12545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:24.689612 12545 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:24.689857 12545 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:24.690177 12545 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:24.690210 12545 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.690239 12545 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:24.690258 12545 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.695860 12545 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:37167
I20260925 14:58:24.695907 12672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:37167 every 8 connection(s)
I20260925 14:58:24.696184 12545 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:24.703346 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12545
I20260925 14:58:24.703447 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:24.703680 8247 external_mini_cluster.cc:1428] Setting key 7c4ed42d52d43bdeb4c9230d52a0e9e6
I20260925 14:58:24.705531 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.697004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:38135 (local address 127.8.13.254:43997)
0925 14:58:24.697082 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.697085 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.697218 (+ 133us) server_negotiation.cc:407] Connection header received
0925 14:58:24.697696 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.697697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.697707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.697757 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:24.698683 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.699180 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.700236 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.700411 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.700454 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.702687 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:24.702689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:24.702689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:24.702697 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:24.704218 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.704627 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.704628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.704629 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.704699 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.704922 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.704923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.704924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.705068 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:24.705137 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.705355 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.705436 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260925 14:58:24.705698 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.696883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43997 (local address 127.8.13.194:38135)
0925 14:58:24.697081 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.697134 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.697617 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.697816 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.697819 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.698087 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:24.698599 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.698605 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.699258 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.699259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.700164 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.700167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.700465 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.700466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.700564 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.701130 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:24.701138 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:24.702579 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:24.704293 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.704298 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.704300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.704561 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.704777 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.704778 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.704778 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.704872 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.705125 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:24.705128 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:24.705242 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.705319 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.705436 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
I20260925 14:58:24.706228 12673 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43997
I20260925 14:58:24.706346 12673 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:24.706517 12673 heartbeater.cc:511] Master 127.8.13.254:43997 requested a full tablet report, sending...
I20260925 14:58:24.706897 12350 ts_manager.cc:194] Registered new tserver with Master: 3f7020ff58624728a75b29c84c18a8ba (127.8.13.194:37167)
I20260925 14:58:24.707274 12350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:38135
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:24.726658 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:36009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:24.807005 12678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:24.807232 12678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:24.807257 12678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:24.808467 12678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:24.808514 12678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:24.808542 12678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:24.810077 12678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:36009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:24.810293 12678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:24.810492 12678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:24.815292 12684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.815299 12686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:24.815307 12683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:24.815586 12678 server_base.cc:1034] running on GCE node
I20260925 14:58:24.815794 12678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:24.816015 12678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:24.817184 12678 hybrid_clock.cc:648] HybridClock initialized: now 1790348304817163 us; error 27 us; skew 500 ppm
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:24.819928 12678 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:24.820286 12678 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:24.820338 12678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:24.820442 12678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:24.821530 12678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "4ee4ee58696b4be5b09f43cfba788058"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "cf52398e880ca3e9fa991ab81a40b4c0"
server_key_iv: "719e4ddd8eb3dee6643bd5d50baa9126"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.821846 12678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "4ee4ee58696b4be5b09f43cfba788058"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "cf52398e880ca3e9fa991ab81a40b4c0"
server_key_iv: "719e4ddd8eb3dee6643bd5d50baa9126"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.823181 12678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:24.823945 12692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.824131 12678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.824219 12678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "4ee4ee58696b4be5b09f43cfba788058"
format_stamp: "Formatted at 2026-09-25 14:58:24 on dist-test-slave-15hz"
server_key: "cf52398e880ca3e9fa991ab81a40b4c0"
server_key_iv: "719e4ddd8eb3dee6643bd5d50baa9126"
server_key_version: "encryptionkey@0"
I20260925 14:58:24.824292 12678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:24.861467 12678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:24.863351 12678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:24.863490 12678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:24.863761 12678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:24.864136 12678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:24.864174 12678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.864197 12678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:24.864207 12678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:24.869710 12678 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:39555
I20260925 14:58:24.869771 12805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:39555 every 8 connection(s)
I20260925 14:58:24.870021 12678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:24.871696 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12678
I20260925 14:58:24.871814 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:24.872022 8247 external_mini_cluster.cc:1428] Setting key e57813a4a22689c3d0b33092306a9eea
Sep 25 14:58:24 dist-test-slave-15hz krb5kdc[12306](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348304, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:24.878995 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.870700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:43051 (local address 127.8.13.254:43997)
0925 14:58:24.870742 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.870744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.870947 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:58:24.871400 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.871401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.871411 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.871458 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:24.872558 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.873134 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.874141 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.874345 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.874400 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.876480 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:24.876482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:24.876482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:24.876489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:24.877762 (+ 1273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.878126 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.878127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.878127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.878180 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.878392 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.878393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.878394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.878503 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:24.878558 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.878824 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.878902 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:24.879186 12808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.870689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43997 (local address 127.8.13.195:43051)
0925 14:58:24.870826 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.870861 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.871322 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.871514 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.871517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.871876 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:24.872474 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.872481 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.873226 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.873227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.874031 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.874033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.874412 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.874413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.874534 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.874962 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:24.874969 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:24.876398 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:24.877837 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.877843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.877845 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.878055 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.878232 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:24.878232 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:24.878233 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:24.878337 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:24.878558 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:24.878565 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:24.878706 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.878796 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.878959 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260925 14:58:24.879618 12806 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43997
I20260925 14:58:24.879706 12806 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:24.879832 12806 heartbeater.cc:511] Master 127.8.13.254:43997 requested a full tablet report, sending...
I20260925 14:58:24.880149 12350 ts_manager.cc:194] Registered new tserver with Master: 4ee4ee58696b4be5b09f43cfba788058 (127.8.13.195:39555)
I20260925 14:58:24.880522 12350 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:43051
I20260925 14:58:24.883509 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:24.890053 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.884725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.8.13.254:43997)
0925 14:58:24.884782 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.884784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.884865 (+ 81us) server_negotiation.cc:407] Connection header received
0925 14:58:24.884921 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.884921 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.884932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.884976 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:24.885503 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.885987 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.886685 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.886880 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.886931 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.887962 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:24.887963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:24.887963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:24.887971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:24.889045 (+ 1074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.889335 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.889336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.889336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.889390 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:24.889536 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:24.889537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:24.889537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:24.889688 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:24.889775 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.889887 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.889963 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:58:24.891330 12350 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:24.893075 12350 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:24.898751 12817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39555 (local address 127.0.0.1:56132)
0925 14:58:24.895280 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.895320 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.895383 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.895588 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.895591 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.895593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:24.895664 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.895666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.896659 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.896660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.897552 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.897554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.898446 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.898447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.898523 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.898538 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.898620 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.898672 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":51,"threads_started":1}
I20260925 14:58:24.898762 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56132 (local address 127.8.13.195:39555)
0925 14:58:24.895367 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.895372 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.895382 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:24.895472 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.895475 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.895490 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.895584 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:24.895726 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.896577 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.897638 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.898367 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.898434 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.898491 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.898637 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.898680 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":70,"threads_started":1}
I20260925 14:58:24.899434 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:39815 (local address 127.0.0.1:37916)
0925 14:58:24.895509 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.895519 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.895752 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.896009 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.896010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.896012 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:24.896063 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.896064 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.897323 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.897324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.898453 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.898456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.899242 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.899243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.899290 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.899293 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.899340 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.899367 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"mutex_wait_us":38,"thread_start_us":79,"threads_started":1}
I20260925 14:58:24.899473 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37916 (local address 127.8.13.193:39815)
0925 14:58:24.895522 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.895546 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.895728 (+ 182us) server_negotiation.cc:407] Connection header received
0925 14:58:24.895918 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.895921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.895937 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.896010 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:24.896128 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.897216 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.898549 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.899169 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.899241 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.899303 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.899355 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.899400 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":66,"threads_started":1}
I20260925 14:58:24.899623 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.8.13.194:37167)
0925 14:58:24.895611 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:24.895630 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:24.895640 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:24.895708 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:24.895711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:24.895726 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:24.895800 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:24.896233 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.897258 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.898345 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.899040 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.899113 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.899168 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:24.899476 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 14:58:24.899520 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":62,"threads_started":1}
I20260925 14:58:24.899434 12818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:24.895151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37167 (local address 127.0.0.1:57740)
0925 14:58:24.895577 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:24.895585 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:24.895635 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:24.896105 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:24.896106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:24.896108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:24.896165 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.896167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.897346 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.897347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.898273 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:24.898275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:24.899198 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:24.899199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:24.899264 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:24.899269 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:24.899316 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:24.899356 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":8,"thread_start_us":67,"threads_started":1}
I20260925 14:58:24.899847 12740 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0849082e04677a93bfaf7ba702822 (DEFAULT_TABLE table=test-table [id=3d9c181a36304e7bb13c6d79d49da5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:24.900205 12740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0849082e04677a93bfaf7ba702822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:24.900485 12607 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0849082e04677a93bfaf7ba702822 (DEFAULT_TABLE table=test-table [id=3d9c181a36304e7bb13c6d79d49da5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:24.900528 12474 tablet_service.cc:1467] Processing CreateTablet for tablet 6fe0849082e04677a93bfaf7ba702822 (DEFAULT_TABLE table=test-table [id=3d9c181a36304e7bb13c6d79d49da5a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:24.900724 12607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0849082e04677a93bfaf7ba702822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:24.900836 12474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6fe0849082e04677a93bfaf7ba702822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:24.903326 12824 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Bootstrap starting.
I20260925 14:58:24.903838 12825 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Bootstrap starting.
I20260925 14:58:24.903937 12823 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Bootstrap starting.
I20260925 14:58:24.903998 12824 tablet_bootstrap.cc:654] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:24.904503 12823 tablet_bootstrap.cc:654] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:24.904506 12825 tablet_bootstrap.cc:654] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:24.904553 12824 log.cc:824] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:24.905026 12825 log.cc:824] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:24.905032 12823 log.cc:824] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:24.905422 12824 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: No bootstrap required, opened a new log
I20260925 14:58:24.905483 12824 ts_tablet_manager.cc:1397] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 14:58:24.905716 12823 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: No bootstrap required, opened a new log
I20260925 14:58:24.905773 12823 ts_tablet_manager.cc:1397] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:24.905890 12825 tablet_bootstrap.cc:492] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: No bootstrap required, opened a new log
I20260925 14:58:24.905958 12825 ts_tablet_manager.cc:1397] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:24.907316 12823 raft_consensus.cc:348] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.907455 12823 raft_consensus.cc:374] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:24.907486 12823 raft_consensus.cc:729] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f7020ff58624728a75b29c84c18a8ba, State: Initialized, Role: FOLLOWER
I20260925 14:58:24.907394 12824 raft_consensus.cc:348] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.907618 12824 raft_consensus.cc:374] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:24.907672 12824 raft_consensus.cc:729] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ee4ee58696b4be5b09f43cfba788058, State: Initialized, Role: FOLLOWER
I20260925 14:58:24.907681 12823 consensus_queue.cc:260] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.907789 12824 consensus_queue.cc:260] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.907912 12825 raft_consensus.cc:348] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.908044 12673 heartbeater.cc:503] Master 127.8.13.254:43997 was elected leader, sending a full tablet report...
I20260925 14:58:24.908068 12824 ts_tablet_manager.cc:1428] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 14:58:24.908059 12823 ts_tablet_manager.cc:1428] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 14:58:24.908177 12806 heartbeater.cc:503] Master 127.8.13.254:43997 was elected leader, sending a full tablet report...
I20260925 14:58:24.908174 12825 raft_consensus.cc:374] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:24.908219 12825 raft_consensus.cc:729] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e565e23a6cda47c0b0385d9012ce6b3c, State: Initialized, Role: FOLLOWER
I20260925 14:58:24.908353 12825 consensus_queue.cc:260] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:24.908871 12540 heartbeater.cc:503] Master 127.8.13.254:43997 was elected leader, sending a full tablet report...
I20260925 14:58:24.908951 12825 ts_tablet_manager.cc:1428] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260925 14:58:24.946728 12674 tablet.cc:2369] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:25.038087 12541 tablet.cc:2369] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:25.118348 12829 raft_consensus.cc:482] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:25.118495 12829 raft_consensus.cc:504] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:25.118825 12829 leader_election.cc:288] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ee4ee58696b4be5b09f43cfba788058 (127.8.13.195:39555), e565e23a6cda47c0b0385d9012ce6b3c (127.8.13.193:39815)
W20260925 14:58:25.120525 12807 tablet.cc:2369] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:25.122352 12822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.119224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:39441 (local address 127.8.13.193:39815)
0925 14:58:25.119278 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.119280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.119324 (+ 44us) server_negotiation.cc:407] Connection header received
0925 14:58:25.119383 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.119384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.119397 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.119466 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:25.119626 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.120156 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.121448 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.121984 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.122064 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.122074 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.122246 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.122282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:25.122351 12820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.119082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:34813 (local address 127.8.13.195:39555)
0925 14:58:25.119132 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.119135 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.119147 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:58:25.119228 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.119229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.119242 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.119308 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:25.119473 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.120022 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.121426 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.122007 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.122066 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.122073 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.122246 (+ 173us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.122282 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:25.122385 12675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.119005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39555 (local address 127.8.13.194:34813)
0925 14:58:25.119074 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.119086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.119136 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.119324 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.119325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.119328 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:25.119405 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:25.119408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.120098 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.120099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:25.121312 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:25.121314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.122083 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.122084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:25.122169 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.122173 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.122236 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.122294 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":2}
I20260925 14:58:25.122394 12832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.119008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:39815 (local address 127.8.13.194:39441)
0925 14:58:25.119245 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.119254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.119330 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.119473 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.119474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.119476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:25.119569 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:25.119571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.120253 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.120254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:25.121371 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:25.121373 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.122083 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.122084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:25.122171 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.122173 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.122236 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.122294 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":85,"thread_start_us":113,"threads_started":1}
I20260925 14:58:25.122725 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0849082e04677a93bfaf7ba702822" candidate_uuid: "3f7020ff58624728a75b29c84c18a8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee4ee58696b4be5b09f43cfba788058" is_pre_election: true
I20260925 14:58:25.122736 12494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0849082e04677a93bfaf7ba702822" candidate_uuid: "3f7020ff58624728a75b29c84c18a8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" is_pre_election: true
I20260925 14:58:25.122859 12760 raft_consensus.cc:2454] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f7020ff58624728a75b29c84c18a8ba in term 0.
I20260925 14:58:25.122862 12494 raft_consensus.cc:2454] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f7020ff58624728a75b29c84c18a8ba in term 0.
I20260925 14:58:25.123050 12563 leader_election.cc:302] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f7020ff58624728a75b29c84c18a8ba, e565e23a6cda47c0b0385d9012ce6b3c; no voters:
I20260925 14:58:25.123196 12829 raft_consensus.cc:2781] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:25.123250 12829 raft_consensus.cc:482] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:25.123268 12829 raft_consensus.cc:3037] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.123980 12829 raft_consensus.cc:504] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:25.124107 12829 leader_election.cc:288] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [CANDIDATE]: Term 1 election: Requested vote from peers 4ee4ee58696b4be5b09f43cfba788058 (127.8.13.195:39555), e565e23a6cda47c0b0385d9012ce6b3c (127.8.13.193:39815)
I20260925 14:58:25.124250 12760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0849082e04677a93bfaf7ba702822" candidate_uuid: "3f7020ff58624728a75b29c84c18a8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ee4ee58696b4be5b09f43cfba788058"
I20260925 14:58:25.124259 12494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6fe0849082e04677a93bfaf7ba702822" candidate_uuid: "3f7020ff58624728a75b29c84c18a8ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e565e23a6cda47c0b0385d9012ce6b3c"
I20260925 14:58:25.124325 12760 raft_consensus.cc:3037] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.124326 12494 raft_consensus.cc:3037] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.125010 12494 raft_consensus.cc:2454] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f7020ff58624728a75b29c84c18a8ba in term 1.
I20260925 14:58:25.125018 12760 raft_consensus.cc:2454] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f7020ff58624728a75b29c84c18a8ba in term 1.
I20260925 14:58:25.125185 12563 leader_election.cc:302] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f7020ff58624728a75b29c84c18a8ba, e565e23a6cda47c0b0385d9012ce6b3c; no voters:
I20260925 14:58:25.125262 12829 raft_consensus.cc:2781] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:25.125411 12829 raft_consensus.cc:686] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 1 LEADER]: Becoming Leader. State: Replica: 3f7020ff58624728a75b29c84c18a8ba, State: Running, Role: LEADER
I20260925 14:58:25.125497 12829 consensus_queue.cc:237] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } }
I20260925 14:58:25.126353 12349 catalog_manager.cc:5495] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f7020ff58624728a75b29c84c18a8ba (127.8.13.194). New cstate: current_term: 1 leader_uuid: "3f7020ff58624728a75b29c84c18a8ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f7020ff58624728a75b29c84c18a8ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37167 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:25.180984 12821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.178714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57742 (local address 127.8.13.194:37167)
0925 14:58:25.178769 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.178771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.178779 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:25.178816 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.178817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.178835 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.178895 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:25.179016 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.179490 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.180271 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:25.180472 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:25.180525 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:25.180675 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:25.180780 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:25.180850 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.180869 (+ 19us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.180892 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:25.182668 12760 raft_consensus.cc:1261] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 1 FOLLOWER]: Refusing update from remote peer 3f7020ff58624728a75b29c84c18a8ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:25.182669 12494 raft_consensus.cc:1261] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 1 FOLLOWER]: Refusing update from remote peer 3f7020ff58624728a75b29c84c18a8ba: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:25.183001 12829 consensus_queue.cc:1035] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "e565e23a6cda47c0b0385d9012ce6b3c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 39815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:25.183139 12833 consensus_queue.cc:1035] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ee4ee58696b4be5b09f43cfba788058" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:25.185467 12836 mvcc.cc:205] Tried to move back new op lower bound from 7333266658025521152 to 7333266657794699264. Current Snapshot: MvccSnapshot[applied={T|T < 7333266658025521152}]
I20260925 14:58:25.186435 12838 mvcc.cc:205] Tried to move back new op lower bound from 7333266658025521152 to 7333266657794699264. Current Snapshot: MvccSnapshot[applied={T|T < 7333266658025521152 or (T in {7333266658025521152})}]
I20260925 14:58:25.187636 12349 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.187742 12349 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.188485 12349 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3d567740939a41849fc239a9a373d4af: Sending DeleteTablet for 3 replicas of tablet 6fe0849082e04677a93bfaf7ba702822
I20260925 14:58:25.188791 12607 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0849082e04677a93bfaf7ba702822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:57740
I20260925 14:58:25.188800 12740 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0849082e04677a93bfaf7ba702822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:56132
I20260925 14:58:25.189002 12846 tablet_replica.cc:330] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: stopping tablet replica
I20260925 14:58:25.189002 12474 tablet_service.cc:1514] Processing DeleteTablet for tablet 6fe0849082e04677a93bfaf7ba702822 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:37916
I20260925 14:58:25.189117 12846 raft_consensus.cc:2229] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.189205 12847 tablet_replica.cc:330] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c: stopping tablet replica
I20260925 14:58:25.189313 12846 raft_consensus.cc:2258] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.189352 12847 raft_consensus.cc:2229] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.189482 12847 raft_consensus.cc:2258] T 6fe0849082e04677a93bfaf7ba702822 P e565e23a6cda47c0b0385d9012ce6b3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.189482 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12412
I20260925 14:58:25.190187 12846 ts_tablet_manager.cc:1905] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.192075 12846 ts_tablet_manager.cc:1918] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:25.192235 12846 log.cc:1196] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/6fe0849082e04677a93bfaf7ba702822
I20260925 14:58:25.192530 12846 ts_tablet_manager.cc:1939] T 6fe0849082e04677a93bfaf7ba702822 P 4ee4ee58696b4be5b09f43cfba788058: Deleting consensus metadata
I20260925 14:58:25.192919 12335 catalog_manager.cc:4841] TS 4ee4ee58696b4be5b09f43cfba788058 (127.8.13.195:39555): tablet 6fe0849082e04677a93bfaf7ba702822 (table test-table [id=3d9c181a36304e7bb13c6d79d49da5a0]) successfully deleted
I20260925 14:58:25.196079 12845 tablet_replica.cc:330] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba: stopping tablet replica
W20260925 14:58:25.196235 12337 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:39815 (error 108)
I20260925 14:58:25.196305 12845 raft_consensus.cc:2229] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:25.196486 12845 raft_consensus.cc:2258] T 6fe0849082e04677a93bfaf7ba702822 P 3f7020ff58624728a75b29c84c18a8ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.196772 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.196640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:39815 (local address 127.0.0.1:37932)
0925 14:58:25.196696 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.196735 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:39815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 14:58:25.196776 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12545
W20260925 14:58:25.197005 12337 catalog_manager.cc:4584] TS e565e23a6cda47c0b0385d9012ce6b3c (127.8.13.193:39815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6fe0849082e04677a93bfaf7ba702822: Network error: Client connection negotiation failed: client connection to 127.8.13.193:39815: connect: Connection refused (error 111)
I20260925 14:58:25.201880 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12678
I20260925 14:58:25.202167 12819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.202037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37167 (local address 127.0.0.1:57756)
0925 14:58:25.202101 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.202127 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:37167: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:58:25.205999 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12319
2026-09-25T14:58:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2006 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-25T14:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:25Z Disabled control of system clock
I20260925 14:58:25.231786 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46589
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:25.311422 12852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:25.311671 12852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:25.311694 12852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:25.312822 12852 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:25.312862 12852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:25.312872 12852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:25.312883 12852 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:25.312893 12852 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:25.314208 12852 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46589
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:25.314435 12852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:25.314692 12852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:25.319242 12860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.319248 12857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.319221 12858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:25.319310 12852 server_base.cc:1034] running on GCE node
I20260925 14:58:25.319727 12852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:25.320015 12852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:25.321161 12852 hybrid_clock.cc:648] HybridClock initialized: now 1790348305321144 us; error 30 us; skew 500 ppm
I20260925 14:58:25.322356 12852 webserver.cc:460] Webserver started at http://127.8.13.254:42839/ using document root <none> and password file <none>
I20260925 14:58:25.322599 12852 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:25.322649 12852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:25.322783 12852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:25.323961 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "92b00ee754fd416da8cd0446dff3ba44"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "aba5a498fd5a9fe7ef5efba44f3d9742"
server_key_iv: "8f93e117ad8d67f7c563a79725f292a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.324318 12852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "92b00ee754fd416da8cd0446dff3ba44"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "aba5a498fd5a9fe7ef5efba44f3d9742"
server_key_iv: "8f93e117ad8d67f7c563a79725f292a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.325547 12852 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:58:25.326191 12866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.326334 12852 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.326419 12852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "92b00ee754fd416da8cd0446dff3ba44"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "aba5a498fd5a9fe7ef5efba44f3d9742"
server_key_iv: "8f93e117ad8d67f7c563a79725f292a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.326479 12852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:25.329700 12852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:25.329939 12852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:25.330062 12852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:25.333711 12852 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:46589
I20260925 14:58:25.333774 12918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:46589 every 8 connection(s)
I20260925 14:58:25.334067 12852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:25.334700 12919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:25.335582 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12852
I20260925 14:58:25.335683 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:25.335844 8247 external_mini_cluster.cc:1428] Setting key 818f8eb2d770b5cdc574d18e6517bd68
I20260925 14:58:25.337692 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: Bootstrap starting.
I20260925 14:58:25.338119 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.336359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.8.13.254:46589)
0925 14:58:25.336515 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.336522 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.336546 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:25.336930 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.336936 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.336975 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.337087 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.337708 (+ 621us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.337711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.337711 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.337715 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.337753 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.337762 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.337825 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.337890 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.337937 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":53,"threads_started":1}
I20260925 14:58:25.338336 12919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:25.338581 12919 log.cc:824] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:25.339182 12919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: No bootstrap required, opened a new log
I20260925 14:58:25.340507 12919 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } }
I20260925 14:58:25.340641 12919 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:25.340674 12919 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b00ee754fd416da8cd0446dff3ba44, State: Initialized, Role: FOLLOWER
I20260925 14:58:25.340790 12919 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } }
I20260925 14:58:25.340870 12919 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:25.340910 12919 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:25.340957 12919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.341445 12919 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } }
I20260925 14:58:25.341535 12919 leader_election.cc:302] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b00ee754fd416da8cd0446dff3ba44; no voters:
I20260925 14:58:25.341679 12919 leader_election.cc:288] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:25.341912 12924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:25.341945 12919 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:25.342108 12924 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [term 1 LEADER]: Becoming Leader. State: Replica: 92b00ee754fd416da8cd0446dff3ba44, State: Running, Role: LEADER
I20260925 14:58:25.342259 12924 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } }
I20260925 14:58:25.342584 12925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b00ee754fd416da8cd0446dff3ba44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } } }
I20260925 14:58:25.342680 12925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:25.342602 12926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b00ee754fd416da8cd0446dff3ba44. Latest consensus state: current_term: 1 leader_uuid: "92b00ee754fd416da8cd0446dff3ba44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b00ee754fd416da8cd0446dff3ba44" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46589 } } }
I20260925 14:58:25.342900 12933 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:25.342974 12926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:25.343389 12933 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:25.344835 12933 catalog_manager.cc:1347] Generated new cluster ID: df2622f12dcd48128f088b6406e15168
I20260925 14:58:25.344887 12933 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:25.354715 12933 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:25.354866 12933 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:25.359843 12933 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: Generated new TSK 0
I20260925 14:58:25.360028 12933 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:25.366282 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--builtin_ntp_servers=127.8.13.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:25.449853 12943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:25.450088 12943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:25.450129 12943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:25.451385 12943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:25.451427 12943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:25.451463 12943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:25.452896 12943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:25.453173 12943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:25.453473 12943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:25.458297 12951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.458351 12948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.458341 12949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:25.459044 12943 server_base.cc:1034] running on GCE node
I20260925 14:58:25.459259 12943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:25.459514 12943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:25.460711 12943 hybrid_clock.cc:648] HybridClock initialized: now 1790348305460687 us; error 39 us; skew 500 ppm
I20260925 14:58:25.461943 12943 webserver.cc:460] Webserver started at http://127.8.13.193:34231/ using document root <none> and password file <none>
I20260925 14:58:25.462170 12943 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:25.462214 12943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:25.462332 12943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:25.463366 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "e0f7465c41721d4144911307e4950997"
server_key_iv: "9bf772ec2562953d51fa44099f6f28ad"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.463726 12943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "e0f7465c41721d4144911307e4950997"
server_key_iv: "9bf772ec2562953d51fa44099f6f28ad"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.464946 12943 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:25.465641 12957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.465786 12943 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:25.465870 12943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "e0f7465c41721d4144911307e4950997"
server_key_iv: "9bf772ec2562953d51fa44099f6f28ad"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.465937 12943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:25.469185 12943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:25.469445 12943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:25.469543 12943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:25.469739 12943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:25.470021 12943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:25.470052 12943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.470077 12943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:25.470108 12943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.475281 12943 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:33281
I20260925 14:58:25.475330 13070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:33281 every 8 connection(s)
I20260925 14:58:25.475659 12943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:25.478258 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.476348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:40675 (local address 127.8.13.254:46589)
0925 14:58:25.476416 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.476418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.476573 (+ 155us) server_negotiation.cc:407] Connection header received
0925 14:58:25.477018 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.477019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.477024 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.477069 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.478079 (+ 1010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.478080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.478080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.478083 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.478098 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.478102 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.478142 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.478193 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.478204 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 14:58:25.478440 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.476283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46589 (local address 127.8.13.193:40675)
0925 14:58:25.476448 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.476482 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.476940 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.477128 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.477131 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.477139 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.477944 (+ 805us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.477944 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.477972 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.478001 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.478142 (+ 141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.478144 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.478147 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.478196 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.478245 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260925 14:58:25.478868 13071 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46589
I20260925 14:58:25.478945 13071 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:25.479063 13071 heartbeater.cc:511] Master 127.8.13.254:46589 requested a full tablet report, sending...
I20260925 14:58:25.479501 12883 ts_manager.cc:194] Registered new tserver with Master: 5109aba9a7cb40c58f4d5806dc9ec5e2 (127.8.13.193:33281)
I20260925 14:58:25.484537 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 12943
I20260925 14:58:25.484629 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:25.484783 8247 external_mini_cluster.cc:1428] Setting key cadd6c766b58376b6ebb392dcebf23bd
I20260925 14:58:25.485773 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--builtin_ntp_servers=127.8.13.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260925 14:58:25.564502 13074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:25.564754 13074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:25.564779 13074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:25.565986 13074 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:25.566032 13074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:25.566067 13074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:25.567432 13074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13074
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:25.567698 13074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:25.568008 13074 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:25.572880 13080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.572876 13079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:25.572961 13074 server_base.cc:1034] running on GCE node
W20260925 14:58:25.572902 13082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:25.573309 13074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:25.573546 13074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:25.574724 13074 hybrid_clock.cc:648] HybridClock initialized: now 1790348305574708 us; error 29 us; skew 500 ppm
I20260925 14:58:25.576052 13074 webserver.cc:460] Webserver started at http://127.8.13.194:39315/ using document root <none> and password file <none>
I20260925 14:58:25.576263 13074 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:25.576313 13074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:25.576438 13074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:25.577491 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "f8dc8189354b43aaa06c3c9a9039a32c"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "b852b83ef47b7f167296a18515a0bca6"
server_key_iv: "7eb0a4914c1283cecdc6a0ea2a32f6ac"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.577831 13074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "f8dc8189354b43aaa06c3c9a9039a32c"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "b852b83ef47b7f167296a18515a0bca6"
server_key_iv: "7eb0a4914c1283cecdc6a0ea2a32f6ac"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.579018 13074 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:25.579681 13088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.579813 13074 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:25.579893 13074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "f8dc8189354b43aaa06c3c9a9039a32c"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "b852b83ef47b7f167296a18515a0bca6"
server_key_iv: "7eb0a4914c1283cecdc6a0ea2a32f6ac"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.579960 13074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:25.583132 13074 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:25.583391 13074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:25.583508 13074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:25.583726 13074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:25.584012 13074 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:25.584045 13074 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.584072 13074 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:25.584095 13074 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.589375 13074 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:41809
I20260925 14:58:25.589411 13201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:41809 every 8 connection(s)
I20260925 14:58:25.589731 13074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:25.590096 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13074
I20260925 14:58:25.590193 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:25.590349 8247 external_mini_cluster.cc:1428] Setting key 92789214de51553c58bc8baf3f8a968c
I20260925 14:58:25.591250 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--builtin_ntp_servers=127.8.13.212:34719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260925 14:58:25.593075 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.590490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:47997 (local address 127.8.13.254:46589)
0925 14:58:25.590540 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.590542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.590796 (+ 254us) server_negotiation.cc:407] Connection header received
0925 14:58:25.591423 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.591424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.591431 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.591457 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.592836 (+ 1379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.592837 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.592838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.592843 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.592863 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.592867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.592931 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.592984 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.592998 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:58:25.593298 13204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.590477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46589 (local address 127.8.13.194:47997)
0925 14:58:25.590674 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.590716 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.591299 (+ 583us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.591669 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.591673 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.591682 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.592682 (+ 1000us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.592682 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.592717 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.592747 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.592932 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.592935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.592937 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.592989 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.593040 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":91,"threads_started":1}
I20260925 14:58:25.593691 13202 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46589
I20260925 14:58:25.593786 13202 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:25.593927 13202 heartbeater.cc:511] Master 127.8.13.254:46589 requested a full tablet report, sending...
I20260925 14:58:25.594259 12883 ts_manager.cc:194] Registered new tserver with Master: f8dc8189354b43aaa06c3c9a9039a32c (127.8.13.194:41809)
W20260925 14:58:25.672210 13205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:25.672418 13205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:25.672441 13205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:25.673602 13205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:25.673645 13205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:25.673671 13205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:25.675005 13205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:34719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46589
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:25.675246 13205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:25.675555 13205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:25.680394 13210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.680372 13211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:25.680352 13213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:25.680509 13205 server_base.cc:1034] running on GCE node
I20260925 14:58:25.680840 13205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:25.681090 13205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:25.682251 13205 hybrid_clock.cc:648] HybridClock initialized: now 1790348305682235 us; error 29 us; skew 500 ppm
I20260925 14:58:25.683594 13205 webserver.cc:460] Webserver started at http://127.8.13.195:46387/ using document root <none> and password file <none>
I20260925 14:58:25.683825 13205 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:25.683874 13205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:25.684000 13205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:25.685110 13205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "4ecbdec243b547aab17d2ddb6bbdeee5"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "871fd155dcc94556c7e849369164f122"
server_key_iv: "e2a3a16ea958bce5945ec4e845074e4b"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.685444 13205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "4ecbdec243b547aab17d2ddb6bbdeee5"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "871fd155dcc94556c7e849369164f122"
server_key_iv: "e2a3a16ea958bce5945ec4e845074e4b"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.686769 13205 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:25.687494 13219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.687650 13205 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:25.687733 13205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "4ecbdec243b547aab17d2ddb6bbdeee5"
format_stamp: "Formatted at 2026-09-25 14:58:25 on dist-test-slave-15hz"
server_key: "871fd155dcc94556c7e849369164f122"
server_key_iv: "e2a3a16ea958bce5945ec4e845074e4b"
server_key_version: "encryptionkey@0"
I20260925 14:58:25.687804 13205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:25.691087 13205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:25.691361 13205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:25.691484 13205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:25.691715 13205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:25.692018 13205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:25.692052 13205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.692080 13205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:25.692104 13205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:25.697451 13205 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:34377
I20260925 14:58:25.697502 13332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:34377 every 8 connection(s)
I20260925 14:58:25.697834 13205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:25.700753 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.698541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:42229 (local address 127.8.13.254:46589)
0925 14:58:25.698602 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.698604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.698836 (+ 232us) server_negotiation.cc:407] Connection header received
0925 14:58:25.699330 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.699331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.699336 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.699391 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.700502 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.700503 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.700503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.700509 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.700524 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.700528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.700579 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.700687 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.700698 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:58:25.700944 13335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.698544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46589 (local address 127.8.13.195:42229)
0925 14:58:25.698721 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.698755 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.699232 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.699454 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.699457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.699466 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.700353 (+ 887us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.700354 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.700384 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.700415 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.700611 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.700616 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.700619 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.700697 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.700750 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
I20260925 14:58:25.701422 13333 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46589
I20260925 14:58:25.701498 13333 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:25.701617 13333 heartbeater.cc:511] Master 127.8.13.254:46589 requested a full tablet report, sending...
I20260925 14:58:25.701911 12883 ts_manager.cc:194] Registered new tserver with Master: 4ecbdec243b547aab17d2ddb6bbdeee5 (127.8.13.195:34377)
I20260925 14:58:25.706451 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13205
I20260925 14:58:25.706538 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:25.706678 8247 external_mini_cluster.cc:1428] Setting key ad35fb7ff6e36f7cedc2631cbb4edb08
I20260925 14:58:25.707486 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:25.709673 12922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.708767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.8.13.254:46589)
0925 14:58:25.708819 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.708821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.708895 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:58:25.708941 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.708942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.708947 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.708992 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.709490 (+ 498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.709492 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.709492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.709495 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.709509 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.709523 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.709562 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.709593 (+ 31us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.709603 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:25.710961 12883 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:25.712561 12883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:25.716192 13343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.714777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41809 (local address 127.0.0.1:57708)
0925 14:58:25.714889 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.714899 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.714939 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.715288 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.715294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.715301 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.715831 (+ 530us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.715831 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.715847 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.715870 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.716040 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.716045 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.716048 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.716112 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.716125 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":68,"threads_started":1}
I20260925 14:58:25.716331 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.714870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.8.13.194:41809)
0925 14:58:25.715116 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.715121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.715131 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:25.715157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.715159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.715168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.715594 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.715931 (+ 337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.715935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.715936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.715939 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.715972 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.715991 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.716049 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.716226 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.716244 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
I20260925 14:58:25.716445 13341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.714561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34377 (local address 127.0.0.1:38098)
0925 14:58:25.714684 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.714714 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.714772 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.715518 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.715518 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.715520 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.715958 (+ 438us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.715958 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.715962 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.715967 (+ 5us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.716330 (+ 363us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.716331 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.716332 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.716360 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.716373 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":68,"threads_started":1}
I20260925 14:58:25.716496 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.715078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48094 (local address 127.8.13.193:33281)
0925 14:58:25.715229 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.715234 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.715333 (+ 99us) server_negotiation.cc:407] Connection header received
0925 14:58:25.715358 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.715360 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.715370 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.715434 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.716156 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.716163 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.716164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.716167 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.716199 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.716222 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.716272 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.716408 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.716422 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260925 14:58:25.716634 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.714982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33281 (local address 127.0.0.1:48094)
0925 14:58:25.715237 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.715245 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.715288 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.715746 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.715747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.715749 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.716072 (+ 323us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.716073 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.716077 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.716085 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.716295 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.716296 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.716297 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.716335 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.716347 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":61,"threads_started":1}
I20260925 14:58:25.716836 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.714678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38098 (local address 127.8.13.195:34377)
0925 14:58:25.715116 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.715121 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.715131 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:25.715157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.715159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.715168 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.715278 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.716203 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.716207 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.716207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.716210 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.716239 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.716254 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.716731 (+ 477us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.716754 (+ 23us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.716770 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":92,"threads_started":1}
I20260925 14:58:25.717135 13136 tablet_service.cc:1467] Processing CreateTablet for tablet 79aed41866d843719dca29a06f7e94db (DEFAULT_TABLE table=test-table [id=9cf156322a084f72b9b2bbb0c1498201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:25.717522 13136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aed41866d843719dca29a06f7e94db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:25.717428 13267 tablet_service.cc:1467] Processing CreateTablet for tablet 79aed41866d843719dca29a06f7e94db (DEFAULT_TABLE table=test-table [id=9cf156322a084f72b9b2bbb0c1498201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:25.717730 13267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aed41866d843719dca29a06f7e94db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:25.720127 13005 tablet_service.cc:1467] Processing CreateTablet for tablet 79aed41866d843719dca29a06f7e94db (DEFAULT_TABLE table=test-table [id=9cf156322a084f72b9b2bbb0c1498201]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:25.720429 13005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 79aed41866d843719dca29a06f7e94db. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:25.720844 13347 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Bootstrap starting.
I20260925 14:58:25.721048 13348 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Bootstrap starting.
I20260925 14:58:25.721714 13348 tablet_bootstrap.cc:654] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:25.722034 13348 log.cc:824] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:25.722893 13347 tablet_bootstrap.cc:654] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:25.723191 13347 log.cc:824] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:25.723711 13351 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: Bootstrap starting.
I20260925 14:58:25.723899 13348 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: No bootstrap required, opened a new log
I20260925 14:58:25.723971 13348 ts_tablet_manager.cc:1397] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:25.724059 13347 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: No bootstrap required, opened a new log
I20260925 14:58:25.724119 13347 ts_tablet_manager.cc:1397] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:25.724329 13351 tablet_bootstrap.cc:654] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:25.724645 13351 log.cc:824] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:25.725322 13351 tablet_bootstrap.cc:492] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: No bootstrap required, opened a new log
I20260925 14:58:25.725387 13351 ts_tablet_manager.cc:1397] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:25.726542 13347 raft_consensus.cc:348] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.726547 13348 raft_consensus.cc:348] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.726715 13347 raft_consensus.cc:374] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:25.726715 13348 raft_consensus.cc:374] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:25.726747 13348 raft_consensus.cc:729] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8dc8189354b43aaa06c3c9a9039a32c, State: Initialized, Role: FOLLOWER
I20260925 14:58:25.726747 13347 raft_consensus.cc:729] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ecbdec243b547aab17d2ddb6bbdeee5, State: Initialized, Role: FOLLOWER
I20260925 14:58:25.726887 13348 consensus_queue.cc:260] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.726887 13347 consensus_queue.cc:260] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.727145 13348 ts_tablet_manager.cc:1428] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:25.727145 13347 ts_tablet_manager.cc:1428] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:25.727260 13333 heartbeater.cc:503] Master 127.8.13.254:46589 was elected leader, sending a full tablet report...
I20260925 14:58:25.727260 13202 heartbeater.cc:503] Master 127.8.13.254:46589 was elected leader, sending a full tablet report...
I20260925 14:58:25.727903 13351 raft_consensus.cc:348] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.728081 13351 raft_consensus.cc:374] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:25.728132 13351 raft_consensus.cc:729] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5109aba9a7cb40c58f4d5806dc9ec5e2, State: Initialized, Role: FOLLOWER
I20260925 14:58:25.728256 13351 consensus_queue.cc:260] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.728529 13351 ts_tablet_manager.cc:1428] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 14:58:25.728554 13071 heartbeater.cc:503] Master 127.8.13.254:46589 was elected leader, sending a full tablet report...
W20260925 14:58:25.840340 13203 tablet.cc:2369] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:25.930285 13355 raft_consensus.cc:482] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:25.930438 13355 raft_consensus.cc:504] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.930794 13355 leader_election.cc:288] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ecbdec243b547aab17d2ddb6bbdeee5 (127.8.13.195:34377), f8dc8189354b43aaa06c3c9a9039a32c (127.8.13.194:41809)
I20260925 14:58:25.932216 13342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.931172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:55833 (local address 127.8.13.195:34377)
0925 14:58:25.931237 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.931239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.931249 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:25.931278 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.931279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.931284 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.931359 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.931952 (+ 593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.931953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.931954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.931958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.932002 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.932007 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.932061 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.932131 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.932146 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:58:25.932215 13073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.931053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:34377 (local address 127.8.13.193:55833)
0925 14:58:25.931112 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.931125 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.931167 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.931365 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.931365 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.931368 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.931852 (+ 484us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.931853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.931860 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.931872 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.932074 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.932074 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.932075 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.932125 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.932139 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260925 14:58:25.932243 13344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.931268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:53287 (local address 127.8.13.194:41809)
0925 14:58:25.931315 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.931317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.931420 (+ 103us) server_negotiation.cc:407] Connection header received
0925 14:58:25.931448 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.931449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.931454 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.931512 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.932054 (+ 542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.932055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.932056 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.932059 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.932075 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.932079 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.932126 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.932178 (+ 52us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.932189 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:58:25.932246 13356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.931165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41809 (local address 127.8.13.193:53287)
0925 14:58:25.931343 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.931353 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:25.931407 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:25.931512 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:25.931513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:25.931515 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:25.931963 (+ 448us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:25.931964 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:25.931971 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:25.931984 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:25.932130 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:25.932130 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:25.932131 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:25.932167 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 14:58:25.932177 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260925 14:58:25.932590 13156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aed41866d843719dca29a06f7e94db" candidate_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" is_pre_election: true
I20260925 14:58:25.932675 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aed41866d843719dca29a06f7e94db" candidate_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" is_pre_election: true
I20260925 14:58:25.932760 13156 raft_consensus.cc:2454] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5109aba9a7cb40c58f4d5806dc9ec5e2 in term 0.
I20260925 14:58:25.932799 13287 raft_consensus.cc:2454] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5109aba9a7cb40c58f4d5806dc9ec5e2 in term 0.
I20260925 14:58:25.932996 12958 leader_election.cc:302] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5109aba9a7cb40c58f4d5806dc9ec5e2, f8dc8189354b43aaa06c3c9a9039a32c; no voters:
I20260925 14:58:25.933111 13355 raft_consensus.cc:2781] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:25.933166 13355 raft_consensus.cc:482] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:25.933192 13355 raft_consensus.cc:3037] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.933872 13355 raft_consensus.cc:504] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.933987 13355 leader_election.cc:288] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [CANDIDATE]: Term 1 election: Requested vote from peers 4ecbdec243b547aab17d2ddb6bbdeee5 (127.8.13.195:34377), f8dc8189354b43aaa06c3c9a9039a32c (127.8.13.194:41809)
I20260925 14:58:25.934162 13287 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aed41866d843719dca29a06f7e94db" candidate_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5"
I20260925 14:58:25.934204 13156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "79aed41866d843719dca29a06f7e94db" candidate_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8dc8189354b43aaa06c3c9a9039a32c"
I20260925 14:58:25.934252 13287 raft_consensus.cc:3037] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.934270 13156 raft_consensus.cc:3037] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:25.935145 13156 raft_consensus.cc:2454] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5109aba9a7cb40c58f4d5806dc9ec5e2 in term 1.
I20260925 14:58:25.935145 13287 raft_consensus.cc:2454] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5109aba9a7cb40c58f4d5806dc9ec5e2 in term 1.
I20260925 14:58:25.935323 12958 leader_election.cc:302] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5109aba9a7cb40c58f4d5806dc9ec5e2, f8dc8189354b43aaa06c3c9a9039a32c; no voters:
I20260925 14:58:25.935424 13355 raft_consensus.cc:2781] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:25.935599 13355 raft_consensus.cc:686] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [term 1 LEADER]: Becoming Leader. State: Replica: 5109aba9a7cb40c58f4d5806dc9ec5e2, State: Running, Role: LEADER
I20260925 14:58:25.935701 13355 consensus_queue.cc:237] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } }
I20260925 14:58:25.936468 12883 catalog_manager.cc:5495] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5109aba9a7cb40c58f4d5806dc9ec5e2 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5109aba9a7cb40c58f4d5806dc9ec5e2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33281 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:25.942879 13346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.942078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48096 (local address 127.8.13.193:33281)
0925 14:58:25.942118 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:25.942120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:25.942145 (+ 25us) server_negotiation.cc:407] Connection header received
0925 14:58:25.942173 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:25.942174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:25.942202 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:25.942236 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:25.942714 (+ 478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:25.942715 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:25.942715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:25.942719 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:25.942733 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:25.942737 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:25.942777 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:25.942809 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 14:58:25.942820 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:25.944808 13156 raft_consensus.cc:1261] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 1 FOLLOWER]: Refusing update from remote peer 5109aba9a7cb40c58f4d5806dc9ec5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:25.944814 13287 raft_consensus.cc:1261] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 1 FOLLOWER]: Refusing update from remote peer 5109aba9a7cb40c58f4d5806dc9ec5e2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:25.945144 13355 consensus_queue.cc:1035] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8dc8189354b43aaa06c3c9a9039a32c" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:25.945290 13357 consensus_queue.cc:1035] T 79aed41866d843719dca29a06f7e94db P 5109aba9a7cb40c58f4d5806dc9ec5e2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ecbdec243b547aab17d2ddb6bbdeee5" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:25.948093 13362 mvcc.cc:205] Tried to move back new op lower bound from 7333266661147328512 to 7333266661113167872. Current Snapshot: MvccSnapshot[applied={T|T < 7333266661147328512}]
I20260925 14:58:25.949766 12883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.949851 12883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:25.950551 12883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 92b00ee754fd416da8cd0446dff3ba44: Sending DeleteTablet for 3 replicas of tablet 79aed41866d843719dca29a06f7e94db
I20260925 14:58:25.950963 13136 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aed41866d843719dca29a06f7e94db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='slave'} at 127.0.0.1:57708
I20260925 14:58:25.951010 13005 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aed41866d843719dca29a06f7e94db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='slave'} at 127.0.0.1:48094
I20260925 14:58:25.951068 13267 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aed41866d843719dca29a06f7e94db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='slave'} at 127.0.0.1:38098
I20260925 14:58:25.951107 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12943
I20260925 14:58:25.951169 13135 tablet_service.cc:1514] Processing DeleteTablet for tablet 79aed41866d843719dca29a06f7e94db with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:25 UTC) from {username='slave'} at 127.0.0.1:57708
I20260925 14:58:25.951593 13372 tablet_replica.cc:330] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: stopping tablet replica
I20260925 14:58:25.951665 13370 tablet_replica.cc:330] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: stopping tablet replica
I20260925 14:58:25.951748 13372 raft_consensus.cc:2229] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.951764 13370 raft_consensus.cc:2229] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:25.951858 13370 raft_consensus.cc:2258] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.951858 13372 raft_consensus.cc:2258] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:25.952283 13372 ts_tablet_manager.cc:1905] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.953562 13370 ts_tablet_manager.cc:1905] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:25.953740 13372 ts_tablet_manager.cc:1918] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:25.953830 13372 log.cc:1196] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/79aed41866d843719dca29a06f7e94db
I20260925 14:58:25.954079 13372 ts_tablet_manager.cc:1939] T 79aed41866d843719dca29a06f7e94db P 4ecbdec243b547aab17d2ddb6bbdeee5: Deleting consensus metadata
I20260925 14:58:25.954516 12867 catalog_manager.cc:4841] TS 4ecbdec243b547aab17d2ddb6bbdeee5 (127.8.13.195:34377): tablet 79aed41866d843719dca29a06f7e94db (table test-table [id=9cf156322a084f72b9b2bbb0c1498201]) successfully deleted
I20260925 14:58:25.954993 13370 ts_tablet_manager.cc:1918] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:25.955053 13370 log.cc:1196] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/79aed41866d843719dca29a06f7e94db
I20260925 14:58:25.955240 13370 ts_tablet_manager.cc:1939] T 79aed41866d843719dca29a06f7e94db P f8dc8189354b43aaa06c3c9a9039a32c: Deleting consensus metadata
I20260925 14:58:25.955725 12867 catalog_manager.cc:4841] TS f8dc8189354b43aaa06c3c9a9039a32c (127.8.13.194:41809): tablet 79aed41866d843719dca29a06f7e94db (table test-table [id=9cf156322a084f72b9b2bbb0c1498201]) successfully deleted
W20260925 14:58:25.955854 12867 catalog_manager.cc:4808] TS f8dc8189354b43aaa06c3c9a9039a32c (127.8.13.194:41809): delete failed for tablet 79aed41866d843719dca29a06f7e94db because the tablet was not found. No further retry: Not found: Tablet not found: 79aed41866d843719dca29a06f7e94db
W20260925 14:58:25.956849 12867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:33281 (error 108)
I20260925 14:58:25.956982 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13074
I20260925 14:58:25.957770 13345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:25.957657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33281 (local address 127.0.0.1:48108)
0925 14:58:25.957698 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:25.957739 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:33281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 14:58:25.957901 12867 catalog_manager.cc:4584] TS 5109aba9a7cb40c58f4d5806dc9ec5e2 (127.8.13.193:33281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 79aed41866d843719dca29a06f7e94db: Network error: Client connection negotiation failed: client connection to 127.8.13.193:33281: connect: Connection refused (error 111)
I20260925 14:58:25.962375 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13205
I20260925 14:58:25.966568 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 12852
2026-09-25T14:58:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (759 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260925 14:58:25.984678 8247 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:25.985715 8247 webserver.cc:460] Webserver started at https://127.0.0.1:36499/ using document root <none> and password file <none>
I20260925 14:58:25.985795 8247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:25.985955 8247 webserver.cc:460] Webserver started at http://127.0.0.1:38167/ using document root <none> and password file <none>
2026-09-25T14:58:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:25Z Disabled control of system clock
I20260925 14:58:25.993245 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33859
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:33859
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36499/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/testchainca.cert with env {}
W20260925 14:58:26.078992 13379 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:26.079226 13379 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:26.079252 13379 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:26.080531 13379 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:26.080581 13379 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:26.080592 13379 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:26.080603 13379 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:26.080614 13379 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:26.080626 13379 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:26.080636 13379 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36499/jwks.json
I20260925 14:58:26.082410 13379 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:33859
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36499/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13379
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:26.082652 13379 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:26.082830 13379 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:26.087466 13387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.087476 13384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.087492 13385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:26.087729 13379 server_base.cc:1034] running on GCE node
I20260925 14:58:26.087955 13379 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:26.088220 13379 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:26.089361 13379 hybrid_clock.cc:648] HybridClock initialized: now 1790348306089345 us; error 34 us; skew 500 ppm
I20260925 14:58:26.090546 13379 webserver.cc:460] Webserver started at http://127.8.13.254:43723/ using document root <none> and password file <none>
I20260925 14:58:26.090787 13379 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:26.090829 13379 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:26.090924 13379 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:26.091995 13379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ceeaa0cb8d6b458193b12403dc82b537"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "f1fbc89b49ff92cabf665275ef53337c"
server_key_iv: "f92f1937f30af87cd93138bfb19361b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.092289 13379 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ceeaa0cb8d6b458193b12403dc82b537"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "f1fbc89b49ff92cabf665275ef53337c"
server_key_iv: "f92f1937f30af87cd93138bfb19361b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.093461 13379 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:26.094173 13393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:26.094362 13379 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:26.094450 13379 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ceeaa0cb8d6b458193b12403dc82b537"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "f1fbc89b49ff92cabf665275ef53337c"
server_key_iv: "f92f1937f30af87cd93138bfb19361b5"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.094515 13379 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:26.112879 13379 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:26.113200 13379 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:26.113308 13379 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:26.116958 13379 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:33859
I20260925 14:58:26.117010 13445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:33859 every 8 connection(s)
I20260925 14:58:26.117326 13379 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:26.117848 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13379
I20260925 14:58:26.117926 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:26.118112 8247 external_mini_cluster.cc:1428] Setting key dbd1e2b163d5b8e0954c785fc5791956
I20260925 14:58:26.118300 13446 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:26.121793 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: Bootstrap starting.
I20260925 14:58:26.122460 13446 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:26.122756 13446 log.cc:824] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:26.123588 13446 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: No bootstrap required, opened a new log
I20260925 14:58:26.124264 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.118642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50310 (local address 127.8.13.254:33859)
0925 14:58:26.118854 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.118863 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.118897 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:58:26.119596 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.119607 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.119698 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.119863 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:26.120019 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.121334 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.122155 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.122777 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.122910 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.123581 (+ 671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:26.123585 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:26.123586 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:26.123788 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:26.123830 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:26.123845 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:26.123908 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:26.123966 (+ 58us) server_negotiation.cc:299] Negotiation successful
0925 14:58:26.124050 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260925 14:58:26.125212 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:26.125732 13446 raft_consensus.cc:348] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } }
I20260925 14:58:26.125874 13446 raft_consensus.cc:374] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:26.125902 13446 raft_consensus.cc:729] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ceeaa0cb8d6b458193b12403dc82b537, State: Initialized, Role: FOLLOWER
I20260925 14:58:26.126029 13446 consensus_queue.cc:260] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } }
I20260925 14:58:26.126107 13446 raft_consensus.cc:388] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:26.126137 13446 raft_consensus.cc:482] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:26.126183 13446 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:26.126748 13446 raft_consensus.cc:504] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } }
I20260925 14:58:26.126847 13446 leader_election.cc:302] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ceeaa0cb8d6b458193b12403dc82b537; no voters:
I20260925 14:58:26.127022 13446 leader_election.cc:288] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:26.127087 13451 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:26.127230 13451 raft_consensus.cc:686] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [term 1 LEADER]: Becoming Leader. State: Replica: ceeaa0cb8d6b458193b12403dc82b537, State: Running, Role: LEADER
I20260925 14:58:26.127281 13446 sys_catalog.cc:564] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:26.127321 13451 consensus_queue.cc:237] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } }
I20260925 14:58:26.127718 13453 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ceeaa0cb8d6b458193b12403dc82b537. Latest consensus state: current_term: 1 leader_uuid: "ceeaa0cb8d6b458193b12403dc82b537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } } }
I20260925 14:58:26.127714 13452 sys_catalog.cc:455] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ceeaa0cb8d6b458193b12403dc82b537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ceeaa0cb8d6b458193b12403dc82b537" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33859 } } }
I20260925 14:58:26.127822 13453 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.127828 13452 sys_catalog.cc:458] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537 [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:26.128949 13467 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:26.128993 13467 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:26.129065 13460 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:26.129228 13460 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:26.130599 13460 catalog_manager.cc:1347] Generated new cluster ID: 251e67b4cb9a4247ad88b28603bbed05
I20260925 14:58:26.130650 13460 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:26.179207 13460 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:26.179950 13460 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:26.186586 13460 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ceeaa0cb8d6b458193b12403dc82b537: Generated new TSK 0
I20260925 14:58:26.186810 13460 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:58:26.209640 13476 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260925 14:58:26.209728 13476 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:58:26.210062 13449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.194424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50322 (local address 127.8.13.254:33859)
0925 14:58:26.194485 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.194487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.194575 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:58:26.194646 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.194646 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.194659 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.194743 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:26.194869 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.195594 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.196368 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.196681 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.196756 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.196936 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:26.209814 (+ 12878us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:58:26.209989 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 14:58:26.210129 13449 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50322: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260925 14:58:26.210403 13475 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:26.194330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33859 (local address 127.0.0.1:50322)
0925 14:58:26.194488 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:26.194497 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:26.194572 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:26.194746 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:26.194747 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:26.194748 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:26.194814 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.194816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.195671 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.195672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.196277 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.196280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.196756 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.196757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.196808 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.196830 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:26.209944 (+ 13114us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0925 14:58:26.210052 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.254:33859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.13.254:33859: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36499/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260925 14:58:26.210829 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13379
2026-09-25T14:58:26Z chronyd exiting
I20260925 14:58:26.246510 8247 test_util.cc:175] -----------------------------------------------
I20260925 14:58:26.246620 8247 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790348284423390-8247-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (262 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260925 14:58:26.247654 8247 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:26.248607 8247 webserver.cc:460] Webserver started at https://127.0.0.1:34671/ using document root <none> and password file <none>
I20260925 14:58:26.248665 8247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:26.248950 8247 webserver.cc:460] Webserver started at http://127.0.0.1:36665/ using document root <none> and password file <none>
2026-09-25T14:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:26Z Disabled control of system clock
I20260925 14:58:26.256016 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39991
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:36989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39991
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/testchainca.cert with env {}
W20260925 14:58:26.334992 13482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:26.335224 13482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:26.335256 13482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:26.336479 13482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:26.336522 13482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:26.336537 13482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:26.336552 13482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:26.336567 13482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:26.336581 13482 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:26.336596 13482 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34671/jwks.json
I20260925 14:58:26.338121 13482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:36989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39991
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:26.338377 13482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:26.338577 13482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:26.343247 13490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.343235 13488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.343232 13487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:26.343281 13482 server_base.cc:1034] running on GCE node
I20260925 14:58:26.343689 13482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:26.343950 13482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:26.345124 13482 hybrid_clock.cc:648] HybridClock initialized: now 1790348306345108 us; error 31 us; skew 500 ppm
I20260925 14:58:26.346279 13482 webserver.cc:460] Webserver started at http://127.8.13.254:43037/ using document root <none> and password file <none>
I20260925 14:58:26.346458 13482 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:26.346498 13482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:26.346588 13482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:26.347644 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "d169c68f945531dc8e74321dcb1db11a"
server_key_iv: "6d5919715c47c66f40a51ce17c5c9ef4"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.347949 13482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "d169c68f945531dc8e74321dcb1db11a"
server_key_iv: "6d5919715c47c66f40a51ce17c5c9ef4"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.349157 13482 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:26.349852 13496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:26.350049 13482 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:26.350136 13482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "d169c68f945531dc8e74321dcb1db11a"
server_key_iv: "6d5919715c47c66f40a51ce17c5c9ef4"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.350203 13482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:26.367841 13482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:26.368155 13482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:26.368260 13482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:26.371830 13482 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:39991
I20260925 14:58:26.371865 13548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:39991 every 8 connection(s)
I20260925 14:58:26.372206 13482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:26.373126 13549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:26.375350 13549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5: Bootstrap starting.
I20260925 14:58:26.375944 13549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:26.376194 13549 log.cc:824] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:26.376874 13549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5: No bootstrap required, opened a new log
I20260925 14:58:26.378425 13549 raft_consensus.cc:348] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } }
I20260925 14:58:26.378571 13549 raft_consensus.cc:374] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:26.378618 13549 raft_consensus.cc:729] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca28be476c9f4928b7aa5ccbd1cc87c5, State: Initialized, Role: FOLLOWER
I20260925 14:58:26.378765 13549 consensus_queue.cc:260] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } }
I20260925 14:58:26.378844 13549 raft_consensus.cc:388] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:26.378875 13549 raft_consensus.cc:482] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:26.378924 13549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:26.379437 13549 raft_consensus.cc:504] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } }
I20260925 14:58:26.379560 13549 leader_election.cc:302] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ca28be476c9f4928b7aa5ccbd1cc87c5; no voters:
I20260925 14:58:26.379724 13549 leader_election.cc:288] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:26.379777 13552 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:26.379930 13552 raft_consensus.cc:686] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [term 1 LEADER]: Becoming Leader. State: Replica: ca28be476c9f4928b7aa5ccbd1cc87c5, State: Running, Role: LEADER
I20260925 14:58:26.379969 13549 sys_catalog.cc:564] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:26.380013 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13482
I20260925 14:58:26.380053 13552 consensus_queue.cc:237] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } }
I20260925 14:58:26.380097 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:26.380282 8247 external_mini_cluster.cc:1428] Setting key fb43eca5be7f1bf6a45e1837e1379b30
I20260925 14:58:26.380561 13554 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ca28be476c9f4928b7aa5ccbd1cc87c5. Latest consensus state: current_term: 1 leader_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } } }
I20260925 14:58:26.380635 13554 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.380882 13553 sys_catalog.cc:455] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ca28be476c9f4928b7aa5ccbd1cc87c5" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39991 } } }
I20260925 14:58:26.380949 13553 sys_catalog.cc:458] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.381363 13558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:26.381858 13558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:26.383455 13558 catalog_manager.cc:1347] Generated new cluster ID: da8077fc89ad4781a35247b9d06dda6c
I20260925 14:58:26.383500 13558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:26.386281 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.380770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.8.13.254:39991)
0925 14:58:26.381069 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.381080 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.381239 (+ 159us) server_negotiation.cc:407] Connection header received
0925 14:58:26.381368 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.381376 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.381450 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.381642 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:26.381865 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.383070 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.384238 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.384883 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.384993 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.385631 (+ 638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:26.385634 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:26.385635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:26.385821 (+ 186us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:26.385858 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:26.385871 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:26.385925 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:26.386001 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 14:58:26.386075 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":73,"threads_started":1}
I20260925 14:58:26.387163 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:26.387915 13558 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:26.388458 13558 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:26.394550 13558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ca28be476c9f4928b7aa5ccbd1cc87c5: Generated new TSK 0
I20260925 14:58:26.394731 13558 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260925 14:58:26.416158 13579 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260925 14:58:26.416239 13579 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260925 14:58:26.416612 13561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.401796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37654 (local address 127.8.13.254:39991)
0925 14:58:26.401846 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.401848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.401968 (+ 120us) server_negotiation.cc:407] Connection header received
0925 14:58:26.402049 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.402050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.402062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.402116 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:26.402240 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.402749 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.403667 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.404020 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.404091 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.404332 (+ 241us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:26.416370 (+ 12038us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:58:26.416535 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260925 14:58:26.416682 13561 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37654: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260925 14:58:26.416698 13578 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:26.401707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39991 (local address 127.0.0.1:37654)
0925 14:58:26.401893 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:26.401901 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:26.401963 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:26.402121 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:26.402122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:26.402124 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:26.402181 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.402184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.402832 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.402833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.403574 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.403579 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.404138 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.404140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.404227 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.404234 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:26.416499 (+ 12265us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0925 14:58:26.416606 (+ 107us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.254:39991: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":82,"threads_started":1}
I20260925 14:58:26.416996 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13482
2026-09-25T14:58:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (206 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260925 14:58:26.453604 8247 mini_oidc.cc:152] Starting JWKS server
I20260925 14:58:26.454252 8247 webserver.cc:460] Webserver started at https://127.0.0.1:44361/ using document root <none> and password file <none>
I20260925 14:58:26.454293 8247 mini_oidc.cc:189] Starting OIDC Discovery server
I20260925 14:58:26.454421 8247 webserver.cc:460] Webserver started at http://127.0.0.1:39015/ using document root <none> and password file <none>
2026-09-25T14:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:26Z Disabled control of system clock
I20260925 14:58:26.461269 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39787
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:35031
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39787
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44361/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:58:26.538533 13585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:26.538748 13585 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:26.538777 13585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:26.539935 13585 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:26.539973 13585 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:26.539987 13585 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:26.540004 13585 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:26.540019 13585 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:26.540033 13585 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:26.540051 13585 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44361/jwks.json
I20260925 14:58:26.541388 13585 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35031
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39787
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44361/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13585
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:26.541630 13585 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:26.541891 13585 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:26.546540 13591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.546530 13593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.546530 13590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:26.546891 13585 server_base.cc:1034] running on GCE node
I20260925 14:58:26.547143 13585 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:26.547413 13585 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:26.548542 13585 hybrid_clock.cc:648] HybridClock initialized: now 1790348306548532 us; error 28 us; skew 500 ppm
I20260925 14:58:26.549718 13585 webserver.cc:460] Webserver started at http://127.8.13.254:33347/ using document root <none> and password file <none>
I20260925 14:58:26.549947 13585 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:26.550006 13585 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:26.550132 13585 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:26.551165 13585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "5635605a10544146b5232a237ea70b60"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e5f5dbf2b5e65f1a563ffa914dfb326f"
server_key_iv: "1930f9bae8058ec2ce6fb46d09acb7a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.551476 13585 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "5635605a10544146b5232a237ea70b60"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e5f5dbf2b5e65f1a563ffa914dfb326f"
server_key_iv: "1930f9bae8058ec2ce6fb46d09acb7a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.552697 13585 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:26.553369 13599 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:26.553531 13585 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:26.553611 13585 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "5635605a10544146b5232a237ea70b60"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e5f5dbf2b5e65f1a563ffa914dfb326f"
server_key_iv: "1930f9bae8058ec2ce6fb46d09acb7a3"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.553679 13585 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:26.582823 13585 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:26.583159 13585 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:26.583308 13585 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:26.587257 13585 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:39787
I20260925 14:58:26.587294 13651 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:39787 every 8 connection(s)
I20260925 14:58:26.587658 13585 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:26.588304 13652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:26.590984 13652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60: Bootstrap starting.
I20260925 14:58:26.591593 13652 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:26.591876 13652 log.cc:824] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:26.592554 13652 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60: No bootstrap required, opened a new log
I20260925 14:58:26.594098 13652 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } }
I20260925 14:58:26.594234 13652 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:26.594255 13652 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5635605a10544146b5232a237ea70b60, State: Initialized, Role: FOLLOWER
I20260925 14:58:26.594388 13652 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } }
I20260925 14:58:26.594444 13652 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:26.594472 13652 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:26.594515 13652 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:26.595008 13652 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } }
I20260925 14:58:26.595099 13652 leader_election.cc:302] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5635605a10544146b5232a237ea70b60; no voters:
I20260925 14:58:26.595252 13652 leader_election.cc:288] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:26.595305 13655 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:26.595422 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13585
I20260925 14:58:26.595456 13655 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [term 1 LEADER]: Becoming Leader. State: Replica: 5635605a10544146b5232a237ea70b60, State: Running, Role: LEADER
I20260925 14:58:26.595521 13652 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:26.595561 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:26.595570 13655 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } }
I20260925 14:58:26.595742 8247 external_mini_cluster.cc:1428] Setting key cfdff1d89fcc75307c15d0bb67d11845
I20260925 14:58:26.596045 13656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5635605a10544146b5232a237ea70b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } } }
I20260925 14:58:26.596124 13656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.596501 13657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5635605a10544146b5232a237ea70b60. Latest consensus state: current_term: 1 leader_uuid: "5635605a10544146b5232a237ea70b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5635605a10544146b5232a237ea70b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39787 } } }
I20260925 14:58:26.596629 13657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.596750 13665 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:26.597404 13665 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:26.598887 13665 catalog_manager.cc:1347] Generated new cluster ID: 98ee5c9417574163ada5dacb0a03302a
I20260925 14:58:26.598929 13665 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:26.601677 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.596335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45662 (local address 127.8.13.254:39787)
0925 14:58:26.596612 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.596621 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.596657 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:58:26.596775 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.596782 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.596852 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.596996 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:26.597326 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.598475 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.599608 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.600208 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.600320 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.600994 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:26.600998 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:26.600998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:26.601197 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:26.601238 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:26.601251 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:26.601308 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:26.601394 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:58:26.601469 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
I20260925 14:58:26.602861 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:26.616202 13665 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:26.616690 13665 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:26.619899 13665 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5635605a10544146b5232a237ea70b60: Generated new TSK 0
I20260925 14:58:26.620070 13665 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:26.631263 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.627825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45664 (local address 127.8.13.254:39787)
0925 14:58:26.627878 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.627880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.628022 (+ 142us) server_negotiation.cc:407] Connection header received
0925 14:58:26.628086 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.628086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.628098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.628154 (+ 56us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:26.628268 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.628740 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.629825 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.630002 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.630050 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.630210 (+ 160us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:26.631061 (+ 851us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:58:26.631192 (+ 131us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 14:58:26.631320 13666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45664: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260925 14:58:26.631361 13681 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:26.627759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39787 (local address 127.0.0.1:45664)
0925 14:58:26.627949 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:26.627959 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:26.628028 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:26.628155 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:26.628156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:26.628158 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:26.628222 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.628225 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.628802 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.628803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.629746 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.629748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.630066 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.630067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.630122 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.630128 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:26.631160 (+ 1032us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0925 14:58:26.631259 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.254:39787: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44361/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":86,"threads_started":1}
I20260925 14:58:26.631778 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13585
E20260925 14:58:26.641778 13682 webserver.cc:533] Webserver: sslize failed:
E20260925 14:58:26.641851 13682 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-25T14:58:26Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-25T14:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:26Z Disabled control of system clock
I20260925 14:58:26.675573 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:41075
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:37059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:41075
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/testchainca.cert with env {}
W20260925 14:58:26.755585 13686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:26.755798 13686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:26.755829 13686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:26.757020 13686 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:26.757061 13686 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:26.757076 13686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:26.757091 13686 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:26.757105 13686 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260925 14:58:26.757122 13686 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260925 14:58:26.757139 13686 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260925 14:58:26.758618 13686 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:41075
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:41075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13686
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:26.758857 13686 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:26.759056 13686 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:26.763520 13694 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.763514 13691 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:26.763594 13692 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:26.763758 13686 server_base.cc:1034] running on GCE node
I20260925 14:58:26.763962 13686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:26.764227 13686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:26.765375 13686 hybrid_clock.cc:648] HybridClock initialized: now 1790348306765360 us; error 30 us; skew 500 ppm
I20260925 14:58:26.766556 13686 webserver.cc:460] Webserver started at http://127.8.13.254:44507/ using document root <none> and password file <none>
I20260925 14:58:26.766757 13686 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:26.766805 13686 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:26.766917 13686 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:26.767962 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "a73670d1d3bc4ef0a973d47d1326e000"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e4f48c9e5e230ef1576309f712053012"
server_key_iv: "bd60db19da6cc38b7fd691103d69c362"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.768296 13686 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "a73670d1d3bc4ef0a973d47d1326e000"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e4f48c9e5e230ef1576309f712053012"
server_key_iv: "bd60db19da6cc38b7fd691103d69c362"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.769440 13686 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:26.770131 13700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:26.770314 13686 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:26.770395 13686 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "a73670d1d3bc4ef0a973d47d1326e000"
format_stamp: "Formatted at 2026-09-25 14:58:26 on dist-test-slave-15hz"
server_key: "e4f48c9e5e230ef1576309f712053012"
server_key_iv: "bd60db19da6cc38b7fd691103d69c362"
server_key_version: "encryptionkey@0"
I20260925 14:58:26.770453 13686 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:26.781551 13686 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:26.781854 13686 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:26.781977 13686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:26.785591 13686 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:41075
I20260925 14:58:26.785631 13752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:41075 every 8 connection(s)
I20260925 14:58:26.785938 13686 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:26.786819 13753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:26.789052 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: Bootstrap starting.
I20260925 14:58:26.789566 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13686
I20260925 14:58:26.789662 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:26.789681 13753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:26.789852 8247 external_mini_cluster.cc:1428] Setting key cedea6b4740924db7d4923dd382f1a38
I20260925 14:58:26.789959 13753 log.cc:824] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:26.790778 13753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: No bootstrap required, opened a new log
I20260925 14:58:26.792800 13753 raft_consensus.cc:348] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } }
I20260925 14:58:26.792932 13753 raft_consensus.cc:374] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:26.792953 13753 raft_consensus.cc:729] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a73670d1d3bc4ef0a973d47d1326e000, State: Initialized, Role: FOLLOWER
I20260925 14:58:26.793057 13753 consensus_queue.cc:260] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } }
I20260925 14:58:26.793128 13753 raft_consensus.cc:388] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:26.793148 13753 raft_consensus.cc:482] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:26.793192 13753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:26.793685 13753 raft_consensus.cc:504] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } }
I20260925 14:58:26.793776 13753 leader_election.cc:302] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a73670d1d3bc4ef0a973d47d1326e000; no voters:
I20260925 14:58:26.793928 13753 leader_election.cc:288] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:26.793998 13758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:26.794128 13758 raft_consensus.cc:686] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [term 1 LEADER]: Becoming Leader. State: Replica: a73670d1d3bc4ef0a973d47d1326e000, State: Running, Role: LEADER
I20260925 14:58:26.794193 13753 sys_catalog.cc:564] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:26.794217 13758 consensus_queue.cc:237] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } }
I20260925 14:58:26.794857 13758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a73670d1d3bc4ef0a973d47d1326e000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } } }
I20260925 14:58:26.794934 13758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.795033 13758 sys_catalog.cc:455] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a73670d1d3bc4ef0a973d47d1326e000. Latest consensus state: current_term: 1 leader_uuid: "a73670d1d3bc4ef0a973d47d1326e000" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a73670d1d3bc4ef0a973d47d1326e000" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41075 } } }
I20260925 14:58:26.795079 13758 sys_catalog.cc:458] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:26.795619 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.790355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55198 (local address 127.8.13.254:41075)
0925 14:58:26.790537 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.790546 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.790581 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:58:26.791134 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.791143 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.791221 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.791376 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:26.791632 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.792574 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.793442 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.793888 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.793998 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.794688 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:26.794692 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:26.794693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:26.794894 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:26.794935 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:26.794950 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:26.795005 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:26.795271 (+ 266us) server_negotiation.cc:299] Negotiation successful
0925 14:58:26.795351 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
W20260925 14:58:26.796015 13773 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:26.796064 13773 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:26.796113 13767 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:26.796247 13767 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:26.796568 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 14:58:26.797747 13767 catalog_manager.cc:1347] Generated new cluster ID: 71a6958ee4594303b8029bff1e77be9f
I20260925 14:58:26.797801 13767 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:26.814558 13767 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:26.815076 13767 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:26.822741 13767 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a73670d1d3bc4ef0a973d47d1326e000: Generated new TSK 0
I20260925 14:58:26.822932 13767 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:27.256805 13757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:26.833157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55204 (local address 127.8.13.254:41075)
0925 14:58:26.833207 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:26.833208 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:26.833323 (+ 115us) server_negotiation.cc:407] Connection header received
0925 14:58:26.833365 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:26.833366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:26.833379 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:26.833453 (+ 74us) server_negotiation.cc:226] Negotiated authn=JWT
0925 14:58:26.833550 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.834340 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.835051 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.835376 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.835442 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.835595 (+ 153us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0925 14:58:27.256582 (+420987us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:58:27.256746 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":29}
W20260925 14:58:27.256868 13757 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55204: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260925 14:58:27.256949 13782 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:26.833105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:41075 (local address 127.0.0.1:55204)
0925 14:58:26.833255 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:26.833264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:26.833317 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:26.833450 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:26.833450 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:26.833452 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0925 14:58:26.833502 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.833504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.834403 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.834404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.834985 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:26.834986 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:26.835447 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:26.835447 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:26.835483 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:26.835488 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0925 14:58:27.256722 (+421234us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0925 14:58:27.256838 (+ 116us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.13.254:41075: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":77,"threads_started":1}
I20260925 14:58:27.257270 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13686
2026-09-25T14:58:27Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (605 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (85 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:27 dist-test-slave-15hz krb5kdc[13794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:27 dist-test-slave-15hz krb5kdc[13794](info): set up 2 sockets
Sep 25 14:58:27 dist-test-slave-15hz krb5kdc[13794](info): commencing operation
krb5kdc: starting...
W20260925 14:58:28.375456 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:28.485594 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44227
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44227
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:28.563221 13807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:28.563418 13807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:28.563438 13807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:28.564621 13807 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:28.564662 13807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:28.564675 13807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:28.564718 13807 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:28.564742 13807 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:28.566213 13807 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44227
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:28.566442 13807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:28.566646 13807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:28.571107 13812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.571158 13813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.571146 13815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:28.571285 13807 server_base.cc:1034] running on GCE node
I20260925 14:58:28.571503 13807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:28.571796 13807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:28.572933 13807 hybrid_clock.cc:648] HybridClock initialized: now 1790348308572917 us; error 29 us; skew 500 ppm
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:28.575210 13807 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:28.575651 13807 webserver.cc:460] Webserver started at http://127.8.13.254:37877/ using document root <none> and password file <none>
I20260925 14:58:28.575855 13807 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:28.575899 13807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:28.576020 13807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:28.577003 13807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ac2f70ea0c2d4744aa86a896c91a076d"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "9e4c694443f9b3492664b0893ae9706b"
server_key_iv: "f9dfe7291711ff93c5b7c7ceffe2d672"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.577332 13807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ac2f70ea0c2d4744aa86a896c91a076d"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "9e4c694443f9b3492664b0893ae9706b"
server_key_iv: "f9dfe7291711ff93c5b7c7ceffe2d672"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.578506 13807 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:28.579135 13822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.579286 13807 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:28.579363 13807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ac2f70ea0c2d4744aa86a896c91a076d"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "9e4c694443f9b3492664b0893ae9706b"
server_key_iv: "f9dfe7291711ff93c5b7c7ceffe2d672"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.579429 13807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:28.599124 13807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:28.601285 13807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:28.601421 13807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:28.605113 13807 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44227
I20260925 14:58:28.605159 13874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44227 every 8 connection(s)
I20260925 14:58:28.605482 13807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:28.606316 13875 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:28.608675 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d: Bootstrap starting.
I20260925 14:58:28.609190 13875 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:28.609419 13875 log.cc:824] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:28.610044 13875 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d: No bootstrap required, opened a new log
I20260925 14:58:28.610100 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13807
I20260925 14:58:28.610208 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:28.610383 8247 external_mini_cluster.cc:1428] Setting key b466436e69d399630c4e9aa310c35a41
I20260925 14:58:28.611979 13875 raft_consensus.cc:348] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } }
I20260925 14:58:28.612128 13875 raft_consensus.cc:374] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:28.612164 13875 raft_consensus.cc:729] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac2f70ea0c2d4744aa86a896c91a076d, State: Initialized, Role: FOLLOWER
I20260925 14:58:28.612327 13875 consensus_queue.cc:260] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } }
I20260925 14:58:28.612411 13875 raft_consensus.cc:388] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:28.612449 13875 raft_consensus.cc:482] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:28.612496 13875 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:28.613446 13875 raft_consensus.cc:504] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } }
I20260925 14:58:28.613533 13875 leader_election.cc:302] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ac2f70ea0c2d4744aa86a896c91a076d; no voters:
I20260925 14:58:28.613670 13875 leader_election.cc:288] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:28.613708 13880 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:28.613860 13880 raft_consensus.cc:686] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [term 1 LEADER]: Becoming Leader. State: Replica: ac2f70ea0c2d4744aa86a896c91a076d, State: Running, Role: LEADER
I20260925 14:58:28.613932 13875 sys_catalog.cc:564] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:28.613967 13880 consensus_queue.cc:237] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } }
I20260925 14:58:28.614338 13882 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ac2f70ea0c2d4744aa86a896c91a076d. Latest consensus state: current_term: 1 leader_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } } }
I20260925 14:58:28.614552 13882 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:28.614485 13881 sys_catalog.cc:455] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ac2f70ea0c2d4744aa86a896c91a076d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44227 } } }
I20260925 14:58:28.614792 13881 sys_catalog.cc:458] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:28.614886 13884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:28.615396 13884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:28.616942 13884 catalog_manager.cc:1347] Generated new cluster ID: 4a418900d7d54c0f8bbb88e07c7fd66d
I20260925 14:58:28.616990 13884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:28.621871 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.610872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46064 (local address 127.8.13.254:44227)
0925 14:58:28.611039 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:28.611050 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:28.611119 (+ 69us) server_negotiation.cc:407] Connection header received
0925 14:58:28.611947 (+ 828us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:28.611953 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:28.612002 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:28.612141 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:28.612900 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.613936 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.614920 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.615552 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.615964 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.618279 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:28.618283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:28.618284 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:28.618297 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:28.620213 (+ 1916us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.620571 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.620574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.620574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.620692 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.620860 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.620861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.620862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.621190 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:28.621353 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:28.621423 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 14:58:28.621666 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260925 14:58:28.628474 13884 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:28.629179 13884 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:28.638085 13884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ac2f70ea0c2d4744aa86a896c91a076d: Generated new TSK 0
I20260925 14:58:28.638311 13884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:28.665791 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:28.744773 13901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:28.744987 13901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:28.745010 13901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:28.746194 13901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:28.746235 13901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:28.746261 13901 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:28.747797 13901 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:28.748014 13901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:28.748229 13901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:28.752950 13907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.752981 13906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.752972 13909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:28.753291 13901 server_base.cc:1034] running on GCE node
I20260925 14:58:28.753499 13901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:28.753726 13901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:28.754861 13901 hybrid_clock.cc:648] HybridClock initialized: now 1790348308754845 us; error 27 us; skew 500 ppm
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:28.757325 13901 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:28.757755 13901 webserver.cc:460] Webserver started at http://127.8.13.193:38705/ using document root <none> and password file <none>
I20260925 14:58:28.757942 13901 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:28.757983 13901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:28.758075 13901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:28.759069 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "29385a343e9c4a4c8e7e126844043ed2"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "0306721792c65d7337dbf2f257f5fbd0"
server_key_iv: "b2bbe9f3e82b532e8810b7e1e24668d3"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.759364 13901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "29385a343e9c4a4c8e7e126844043ed2"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "0306721792c65d7337dbf2f257f5fbd0"
server_key_iv: "b2bbe9f3e82b532e8810b7e1e24668d3"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.760661 13901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:28.761329 13916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.761494 13901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:28.761590 13901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "29385a343e9c4a4c8e7e126844043ed2"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "0306721792c65d7337dbf2f257f5fbd0"
server_key_iv: "b2bbe9f3e82b532e8810b7e1e24668d3"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.761649 13901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:28.773731 13901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:28.775489 13901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:28.775671 13901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:28.775873 13901 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:28.776156 13901 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:28.776189 13901 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.776211 13901 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:28.776234 13901 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.781734 13901 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:36761
I20260925 14:58:28.781785 14029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:36761 every 8 connection(s)
I20260925 14:58:28.782179 13901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:28.790409 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 13901
I20260925 14:58:28.790529 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:28.790741 8247 external_mini_cluster.cc:1428] Setting key 292c583db8ec77591df1d8d87ddfd1fa
I20260925 14:58:28.791635 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.782916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57331 (local address 127.8.13.254:44227)
0925 14:58:28.782966 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:28.782967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:28.783211 (+ 244us) server_negotiation.cc:407] Connection header received
0925 14:58:28.784138 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:28.784139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:28.784154 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:28.784224 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:28.784830 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.785359 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.786510 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.786727 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.786786 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.789001 (+ 2215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:28.789002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:28.789003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:28.789011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:28.790176 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.790688 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.790689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.790690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.790745 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.791023 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.791025 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.791025 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.791148 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:28.791206 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:28.791462 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 14:58:28.791569 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:28.791883 14032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.782877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44227 (local address 127.8.13.193:57331)
0925 14:58:28.783050 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:28.783091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:28.784045 (+ 954us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:28.784280 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:28.784284 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:28.784544 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:28.784764 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.784769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.785527 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.785528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.786415 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.786418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.786825 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.786825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.786947 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.787455 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:28.787464 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:28.788908 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:28.790275 (+ 1367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.790285 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.790288 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.790612 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.790804 (+ 192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.790805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.790805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.790944 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.791206 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:28.791212 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:28.791343 (+ 131us) client_negotiation.cc:769] Sending connection context
0925 14:58:28.791444 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 14:58:28.791657 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":99,"threads_started":1}
I20260925 14:58:28.792376 14030 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44227
I20260925 14:58:28.792478 14030 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:28.792675 14030 heartbeater.cc:511] Master 127.8.13.254:44227 requested a full tablet report, sending...
I20260925 14:58:28.793303 13839 ts_manager.cc:194] Registered new tserver with Master: 29385a343e9c4a4c8e7e126844043ed2 (127.8.13.193:36761)
I20260925 14:58:28.794184 13839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:57331
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:28.814108 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:28.901120 14035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:28.901333 14035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:28.901355 14035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:28.902477 14035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:28.902510 14035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:28.902535 14035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:28.904018 14035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:28.904214 14035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:28.904395 14035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:28.909111 14043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.909183 14040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:28.909153 14041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:28.909292 14035 server_base.cc:1034] running on GCE node
I20260925 14:58:28.909502 14035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:28.909754 14035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:28.910908 14035 hybrid_clock.cc:648] HybridClock initialized: now 1790348308910894 us; error 28 us; skew 500 ppm
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:28.913440 14035 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:28.913983 14035 webserver.cc:460] Webserver started at http://127.8.13.194:44903/ using document root <none> and password file <none>
I20260925 14:58:28.914222 14035 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:28.914275 14035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:28.914469 14035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:28.915671 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "21654ca5bfba438ab055db1fddd438d8"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "faec887b08fed5431be7134ff734dd28"
server_key_iv: "8a30e1b808768d4e021c6e6e9111b8a7"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.916016 14035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "21654ca5bfba438ab055db1fddd438d8"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "faec887b08fed5431be7134ff734dd28"
server_key_iv: "8a30e1b808768d4e021c6e6e9111b8a7"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.917320 14035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:28.918037 14050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.918195 14035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:28.918288 14035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "21654ca5bfba438ab055db1fddd438d8"
format_stamp: "Formatted at 2026-09-25 14:58:28 on dist-test-slave-15hz"
server_key: "faec887b08fed5431be7134ff734dd28"
server_key_iv: "8a30e1b808768d4e021c6e6e9111b8a7"
server_key_version: "encryptionkey@0"
I20260925 14:58:28.918351 14035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:28.939332 14035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:28.941809 14035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:28.941941 14035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:28.942159 14035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:28.942454 14035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:28.942487 14035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.942515 14035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:28.942538 14035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:28.947865 14035 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:34981
I20260925 14:58:28.947902 14163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:34981 every 8 connection(s)
I20260925 14:58:28.948211 14035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:28.948714 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14035
I20260925 14:58:28.948820 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:28.949029 8247 external_mini_cluster.cc:1428] Setting key d0c6a25122d4ff6931cd3965dd1ef702
Sep 25 14:58:28 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348308, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:28.957805 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.948908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:60155 (local address 127.8.13.254:44227)
0925 14:58:28.949013 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:28.949016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:28.949329 (+ 313us) server_negotiation.cc:407] Connection header received
0925 14:58:28.950287 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:28.950289 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:28.950301 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:28.950366 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:28.950971 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.951467 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.952462 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.952672 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.952729 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.954925 (+ 2196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:28.954927 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:28.954927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:28.954935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:28.956409 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.956833 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.956833 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.956834 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.956890 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:28.957115 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:28.957116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:28.957116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:28.957269 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:28.957337 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:28.957617 (+ 280us) server_negotiation.cc:299] Negotiation successful
0925 14:58:28.957713 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260925 14:58:28.957991 14166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:28.948884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44227 (local address 127.8.13.194:60155)
0925 14:58:28.949196 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:28.949246 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:28.950183 (+ 937us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:28.950426 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:28.950429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:28.950673 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:28.950885 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.950890 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.951569 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.951570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.952380 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:28.952383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:28.952740 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:28.952741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:28.952855 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:28.953371 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:28.953380 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:28.954809 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:28.956491 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.956501 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.956504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.956754 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.956945 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:28.956945 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:28.956946 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:28.957041 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:28.957340 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:28.957349 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:28.957495 (+ 146us) client_negotiation.cc:769] Sending connection context
0925 14:58:28.957589 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 14:58:28.957758 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260925 14:58:28.958555 14164 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44227
I20260925 14:58:28.958650 14164 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:28.958834 14164 heartbeater.cc:511] Master 127.8.13.254:44227 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
I20260925 14:58:28.959244 13839 ts_manager.cc:194] Registered new tserver with Master: 21654ca5bfba438ab055db1fddd438d8 (127.8.13.194:34981)
I20260925 14:58:28.959676 13839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:60155
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:28.973826 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:29.053339 14169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:29.053561 14169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:29.053586 14169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:29.054801 14169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:29.054849 14169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:29.054876 14169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:29.056406 14169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:29.056623 14169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:29.056813 14169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:29.061524 14175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:29.061617 14177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:29.061663 14174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:29.061760 14169 server_base.cc:1034] running on GCE node
I20260925 14:58:29.061971 14169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:29.062222 14169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:29.063378 14169 hybrid_clock.cc:648] HybridClock initialized: now 1790348309063369 us; error 38 us; skew 500 ppm
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.066103 14169 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:29.066541 14169 webserver.cc:460] Webserver started at http://127.8.13.195:34053/ using document root <none> and password file <none>
I20260925 14:58:29.066756 14169 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:29.066802 14169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:29.066924 14169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:29.067991 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "92b0204353a54c2e8f4ebed82edab04c"
format_stamp: "Formatted at 2026-09-25 14:58:29 on dist-test-slave-15hz"
server_key: "caf8d734729fb23255155cbe461687b8"
server_key_iv: "0472b3b35049afa084095e381889bf0b"
server_key_version: "encryptionkey@0"
I20260925 14:58:29.068308 14169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "92b0204353a54c2e8f4ebed82edab04c"
format_stamp: "Formatted at 2026-09-25 14:58:29 on dist-test-slave-15hz"
server_key: "caf8d734729fb23255155cbe461687b8"
server_key_iv: "0472b3b35049afa084095e381889bf0b"
server_key_version: "encryptionkey@0"
I20260925 14:58:29.069541 14169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:29.070168 14184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:29.070324 14169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:29.070403 14169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "92b0204353a54c2e8f4ebed82edab04c"
format_stamp: "Formatted at 2026-09-25 14:58:29 on dist-test-slave-15hz"
server_key: "caf8d734729fb23255155cbe461687b8"
server_key_iv: "0472b3b35049afa084095e381889bf0b"
server_key_version: "encryptionkey@0"
I20260925 14:58:29.070461 14169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:29.089794 14169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:29.091727 14169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:29.091862 14169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:29.092067 14169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:29.092352 14169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:29.092383 14169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:29.092411 14169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:29.092434 14169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:29.097615 14169 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:37079
I20260925 14:58:29.097688 14297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:37079 every 8 connection(s)
I20260925 14:58:29.097951 14169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:29.098623 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14169
I20260925 14:58:29.098731 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:29.098923 8247 external_mini_cluster.cc:1428] Setting key e0d2fd1e58b598187f3f76946c3cad92
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:29.106787 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:29.098667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:50455 (local address 127.8.13.254:44227)
0925 14:58:29.098725 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.098727 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.098953 (+ 226us) server_negotiation.cc:407] Connection header received
0925 14:58:29.100065 (+ 1112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.100066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.100076 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.100117 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.100692 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.101256 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.102199 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.102382 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.102436 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:29.104463 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:29.104464 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:29.104465 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:29.104470 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:29.105608 (+ 1138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.105990 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.105991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.105991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.106032 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.106213 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.106213 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.106213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.106327 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:29.106374 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:29.106660 (+ 286us) server_negotiation.cc:299] Negotiation successful
0925 14:58:29.106721 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:29.106962 14300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:29.098629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44227 (local address 127.8.13.195:50455)
0925 14:58:29.098783 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:29.098833 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:29.099983 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:29.100177 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:29.100180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:29.100427 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:29.100619 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:29.100623 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.101322 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.101322 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:29.102128 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:29.102130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.102448 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.102449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:29.102558 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:29.103038 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:29.103046 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:29.104393 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:29.105689 (+ 1296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:29.105695 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:29.105697 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:29.105934 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:29.106080 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:29.106080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:29.106081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:29.106172 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:29.106404 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:29.106410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:29.106538 (+ 128us) client_negotiation.cc:769] Sending connection context
0925 14:58:29.106637 (+ 99us) client_negotiation.cc:240] Negotiation successful
0925 14:58:29.106761 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260925 14:58:29.107544 14298 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44227
I20260925 14:58:29.107642 14298 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:29.107791 14298 heartbeater.cc:511] Master 127.8.13.254:44227 requested a full tablet report, sending...
I20260925 14:58:29.108129 13839 ts_manager.cc:194] Registered new tserver with Master: 92b0204353a54c2e8f4ebed82edab04c (127.8.13.195:37079)
I20260925 14:58:29.108487 13839 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:50455
I20260925 14:58:29.110543 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/test-admin.keytab.
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.125283 8247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260925 14:58:29.126729 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:29.126103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.8.13.193:36761)
0925 14:58:29.126241 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.126246 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.126256 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:29.126281 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.126282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.126296 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.126359 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.126652 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
W20260925 14:58:29.126793 8247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.127741 8247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.128495 8247 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:29.135236 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:29.128960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46464 (local address 127.8.13.193:36761)
0925 14:58:29.129013 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.129015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.129024 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:29.129068 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.129068 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.129096 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.129142 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.129453 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.130180 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.130835 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.131155 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.131203 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:29.133011 (+ 1808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:29.133018 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:29.133018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:29.133024 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:29.134409 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.134656 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.134658 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.134659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.134704 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.134818 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.134818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.134819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.134927 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:29.135023 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:29.135058 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 14:58:29.135134 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 14:58:29.136803 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:29.136321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46470 (local address 127.8.13.193:36761)
0925 14:58:29.136358 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.136360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.136390 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:29.136454 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.136454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.136464 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.136502 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.136761 (+ 259us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46470: BlockingRecv error: recv got EOF from 127.0.0.1:46470 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:58:29.136864 8247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.137662 8247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.138463 8247 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:29.144328 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:29.138917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.8.13.193:36761)
0925 14:58:29.138954 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.138955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.138965 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:29.139017 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.139018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.139024 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.139054 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.139351 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.139865 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.140501 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.140698 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.140747 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:29.142475 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:29.142475 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:29.142476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:29.142480 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:29.143498 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.143834 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.143835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.143836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.143882 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.144000 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.144000 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.144001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.144078 (+ 77us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:29.144142 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:29.144211 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:58:29.144266 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:58:29.145583 14305 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:29.145215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46488 (local address 127.8.13.193:36761)
0925 14:58:29.145250 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.145252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.145260 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:29.145294 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.145295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.145304 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.145344 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.145540 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46488: BlockingRecv error: recv got EOF from 127.0.0.1:46488 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 14:58:29.145669 8247 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790348284423390-8247-0/krb5kdc/krb5cc' of type FILE
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.146546 8247 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:29.147303 8247 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[13794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 14:58:29.153406 14305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:29.147799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46502 (local address 127.8.13.193:36761)
0925 14:58:29.147834 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:29.147835 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:29.147858 (+ 23us) server_negotiation.cc:407] Connection header received
0925 14:58:29.147904 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:29.147905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:29.147911 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:29.147952 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:29.148254 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.148702 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.149378 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:29.149558 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:29.149608 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:29.151440 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:29.151442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:29.151442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:29.151447 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:29.152529 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.152818 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.152818 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.152818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.152860 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:29.153080 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:29.153081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:29.153081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:29.153159 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:29.153220 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:29.153299 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 14:58:29.153347 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 14:58:29.153920 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13901
I20260925 14:58:29.158327 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14035
I20260925 14:58:29.162441 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14169
I20260925 14:58:29.165961 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 13807
2026-09-25T14:58:29Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1736 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[14318](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[14318](info): set up 2 sockets
Sep 25 14:58:29 dist-test-slave-15hz krb5kdc[14318](info): commencing operation
krb5kdc: starting...
W20260925 14:58:30.119769 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:30Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab.
Entry for principal oryx/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab.
I20260925 14:58:30.234820 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39483
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab
--principal=oryx/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39483
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.315289 14331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.315522 14331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.315580 14331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.317205 14331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:30.317312 14331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.317355 14331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.317385 14331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:30.317401 14331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:30.319464 14331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39483
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab
--principal=oryx/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:30.319720 14331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.319909 14331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.324573 14339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.324550 14336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.324563 14337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.324772 14331 server_base.cc:1034] running on GCE node
I20260925 14:58:30.325142 14331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.325452 14331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.326624 14331 hybrid_clock.cc:648] HybridClock initialized: now 1790348310326590 us; error 48 us; skew 500 ppm
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:30.335116 14331 init.cc:377] Logged in from keytab as oryx/127.8.13.254@KRBTEST.COM (short username oryx)
I20260925 14:58:30.335665 14331 webserver.cc:460] Webserver started at http://127.8.13.254:40315/ using document root <none> and password file <none>
I20260925 14:58:30.335881 14331 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:30.335923 14331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:30.336023 14331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:30.337109 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "3577c8dc9c9a4d8bbd77055ee1786b16"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "3bea1e88e672b8c7fdd7f13417cd61b4"
server_key_iv: "0507fe953943f98bb6e08b89d9fa8f00"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.337404 14331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "3577c8dc9c9a4d8bbd77055ee1786b16"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "3bea1e88e672b8c7fdd7f13417cd61b4"
server_key_iv: "0507fe953943f98bb6e08b89d9fa8f00"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.338661 14331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:30.339329 14346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.339566 14331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:30.339663 14331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "3577c8dc9c9a4d8bbd77055ee1786b16"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "3bea1e88e672b8c7fdd7f13417cd61b4"
server_key_iv: "0507fe953943f98bb6e08b89d9fa8f00"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.339733 14331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:30.364845 14331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:30.366621 14331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:30.366753 14331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:30.370460 14331 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:39483
I20260925 14:58:30.370512 14398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:39483 every 8 connection(s)
I20260925 14:58:30.370841 14331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:30.371832 14399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:30.374096 14399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Bootstrap starting.
I20260925 14:58:30.374630 14399 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:30.374857 14399 log.cc:824] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:30.375622 14399 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: No bootstrap required, opened a new log
I20260925 14:58:30.377219 14399 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } }
I20260925 14:58:30.377377 14399 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:30.377400 14399 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3577c8dc9c9a4d8bbd77055ee1786b16, State: Initialized, Role: FOLLOWER
I20260925 14:58:30.377528 14399 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } }
I20260925 14:58:30.377604 14399 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:30.377637 14399 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:30.377671 14399 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:30.378469 14399 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } }
I20260925 14:58:30.378562 14399 leader_election.cc:302] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3577c8dc9c9a4d8bbd77055ee1786b16; no voters:
I20260925 14:58:30.378710 14399 leader_election.cc:288] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:30.378764 14402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:30.378930 14402 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 LEADER]: Becoming Leader. State: Replica: 3577c8dc9c9a4d8bbd77055ee1786b16, State: Running, Role: LEADER
I20260925 14:58:30.378957 14399 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:30.379082 14402 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } }
I20260925 14:58:30.379456 14404 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3577c8dc9c9a4d8bbd77055ee1786b16. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } }
I20260925 14:58:30.379482 14403 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } }
I20260925 14:58:30.379593 14404 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:30.379657 14403 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:30.379720 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14331
I20260925 14:58:30.379817 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:30.379999 8247 external_mini_cluster.cc:1428] Setting key 11c034a2cc5892edd7fddb1e3de74b9e
W20260925 14:58:30.380856 14417 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:30.380913 14417 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:30.380954 14411 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:30.381100 14411 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:30.382545 14411 catalog_manager.cc:1347] Generated new cluster ID: 9d73cb0257cd4c7d84a0432090fe0440
I20260925 14:58:30.382592 14411 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:30.392194 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.380695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50726 (local address 127.8.13.254:39483)
0925 14:58:30.380853 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.380862 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.380895 (+ 33us) server_negotiation.cc:407] Connection header received
0925 14:58:30.381027 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.381034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.381097 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.381245 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.381963 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.383386 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.384507 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.385213 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.385312 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.388272 (+ 2960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:30.388276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:30.388277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:30.388292 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:30.390373 (+ 2081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.390795 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.390799 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.390799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.390929 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.391111 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.391112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.391112 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.391453 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:30.391659 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:30.391766 (+ 107us) server_negotiation.cc:299] Negotiation successful
0925 14:58:30.391924 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260925 14:58:30.394435 14411 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:30.394944 14411 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:30.397497 14411 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Generated new TSK 0
I20260925 14:58:30.397650 14411 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
I20260925 14:58:30.426649 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.507429 14425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.507695 14425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.507721 14425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.508919 14425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.508956 14425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.508983 14425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:30.510445 14425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:30.510689 14425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.510903 14425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.515722 14431 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.515710 14433 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.515733 14430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.515846 14425 server_base.cc:1034] running on GCE node
I20260925 14:58:30.516113 14425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.516326 14425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.517491 14425 hybrid_clock.cc:648] HybridClock initialized: now 1790348310517475 us; error 28 us; skew 500 ppm
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:30.520066 14425 init.cc:377] Logged in from keytab as oryx/127.8.13.193@KRBTEST.COM (short username oryx)
I20260925 14:58:30.520532 14425 webserver.cc:460] Webserver started at http://127.8.13.193:33453/ using document root <none> and password file <none>
I20260925 14:58:30.520753 14425 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:30.520816 14425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:30.520932 14425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:30.522013 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "71b2cc4469ea000a06d682c7df3a987f"
server_key_iv: "6e29ee0cccb916864ce9a06326e9b4b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.522346 14425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "71b2cc4469ea000a06d682c7df3a987f"
server_key_iv: "6e29ee0cccb916864ce9a06326e9b4b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.523643 14425 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:30.524396 14440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.524541 14425 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:30.524621 14425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "71b2cc4469ea000a06d682c7df3a987f"
server_key_iv: "6e29ee0cccb916864ce9a06326e9b4b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.524703 14425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:30.532940 14425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:30.535030 14425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:30.535173 14425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:30.535377 14425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:30.535686 14425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:30.535720 14425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.535749 14425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:30.535773 14425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.541352 14425 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:40377
I20260925 14:58:30.541400 14553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:40377 every 8 connection(s)
I20260925 14:58:30.541718 14425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:30.550410 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.542462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:59107 (local address 127.8.13.254:39483)
0925 14:58:30.542528 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.542530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.542731 (+ 201us) server_negotiation.cc:407] Connection header received
0925 14:58:30.543226 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.543227 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.543242 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.543300 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.544292 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.544825 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.545764 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.545954 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.546001 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.548054 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:30.548055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:30.548055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:30.548062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:30.549219 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.549595 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.549597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.549597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.549658 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.549845 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.549845 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.549846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.549950 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:30.550013 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:30.550255 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 14:58:30.550340 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:58:30.550505 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.542456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:59107)
0925 14:58:30.542603 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:30.542653 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:30.543108 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:30.543397 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:30.543402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:30.543995 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:30.544196 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.544201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.544893 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.544894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.545698 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.545700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.546010 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.546011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.546105 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.546601 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:30.546609 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:30.547971 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:30.549288 (+ 1317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.549294 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.549296 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.549513 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.549714 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.549714 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.549715 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.549803 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.550001 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:30.550003 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:30.550119 (+ 116us) client_negotiation.cc:769] Sending connection context
0925 14:58:30.550201 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:58:30.550327 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260925 14:58:30.550973 14554 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:30.551079 14554 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:30.551258 14554 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:30.551316 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14425
I20260925 14:58:30.551420 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:30.551664 8247 external_mini_cluster.cc:1428] Setting key 5b98e66e43c02a202cfca8edf510b255
I20260925 14:58:30.551831 14363 ts_manager.cc:194] Registered new tserver with Master: ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377)
I20260925 14:58:30.552704 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:59107
WARNING: no policy specified for oryx/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
I20260925 14:58:30.576202 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.656092 14559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.656452 14559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.656527 14559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.657928 14559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.658013 14559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.658052 14559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:30.659659 14559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:30.659914 14559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.660264 14559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.665477 14567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.665457 14564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.665454 14565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.665529 14559 server_base.cc:1034] running on GCE node
I20260925 14:58:30.665861 14559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.666090 14559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.667224 14559 hybrid_clock.cc:648] HybridClock initialized: now 1790348310667213 us; error 29 us; skew 500 ppm
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:30.669786 14559 init.cc:377] Logged in from keytab as oryx/127.8.13.194@KRBTEST.COM (short username oryx)
I20260925 14:58:30.670270 14559 webserver.cc:460] Webserver started at http://127.8.13.194:39337/ using document root <none> and password file <none>
I20260925 14:58:30.670467 14559 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:30.670516 14559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:30.670657 14559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:30.671798 14559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "f854b76c9c734857bbf73a2741070116"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "971a00edaaf1d4b272442ce9322bcd00"
server_key_iv: "fdc25a45f5726b734155e075d26c80ab"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.672139 14559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "f854b76c9c734857bbf73a2741070116"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "971a00edaaf1d4b272442ce9322bcd00"
server_key_iv: "fdc25a45f5726b734155e075d26c80ab"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.673434 14559 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:30.674155 14574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.674314 14559 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:30.674400 14559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "f854b76c9c734857bbf73a2741070116"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "971a00edaaf1d4b272442ce9322bcd00"
server_key_iv: "fdc25a45f5726b734155e075d26c80ab"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.674465 14559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:30.705996 14559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:30.708271 14559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:30.708416 14559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:30.708654 14559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:30.708963 14559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:30.708997 14559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.709028 14559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:30.709049 14559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.714622 14559 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45079
I20260925 14:58:30.714660 14687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45079 every 8 connection(s)
I20260925 14:58:30.714973 14559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:30.721396 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14559
I20260925 14:58:30.721510 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:30.721681 8247 external_mini_cluster.cc:1428] Setting key bd302ac780dbfe98586e06c31801e72a
I20260925 14:58:30.725534 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.715808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:54587 (local address 127.8.13.254:39483)
0925 14:58:30.715875 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.715877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.716069 (+ 192us) server_negotiation.cc:407] Connection header received
0925 14:58:30.716534 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.716535 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.716547 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.716617 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.717519 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.718150 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.719122 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.719318 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.719376 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.721456 (+ 2080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:30.721458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:30.721458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:30.721466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:30.724035 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.724562 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.724563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.724564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.724629 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.724850 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.724850 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.724850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.725024 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:30.725089 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:30.725341 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 14:58:30.725438 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:30.725620 14690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.715783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:54587)
0925 14:58:30.715945 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:30.715981 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:30.716451 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:30.716682 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:30.716685 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:30.717230 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:30.717431 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.717436 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.718234 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.718235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.719042 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.719044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.719384 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.719385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.719483 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.720014 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:30.720022 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:30.721362 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:30.724120 (+ 2758us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.724132 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.724135 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.724469 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.724682 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.724683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.724683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.724801 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.725081 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:30.725084 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:30.725214 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 14:58:30.725296 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 14:58:30.725432 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260925 14:58:30.726142 14688 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:30.726243 14688 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:30.726423 14688 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:30.726811 14363 ts_manager.cc:194] Registered new tserver with Master: f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
I20260925 14:58:30.727190 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:54587
WARNING: no policy specified for oryx/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
I20260925 14:58:30.747225 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.828466 14693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.828696 14693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.828722 14693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.829908 14693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.829948 14693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.829975 14693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:30.831548 14693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39483
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:30.831760 14693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.831945 14693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.836700 14699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.836716 14698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.836695 14701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.836966 14693 server_base.cc:1034] running on GCE node
I20260925 14:58:30.837159 14693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.837368 14693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.838521 14693 hybrid_clock.cc:648] HybridClock initialized: now 1790348310838505 us; error 31 us; skew 500 ppm
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:30.841218 14693 init.cc:377] Logged in from keytab as oryx/127.8.13.195@KRBTEST.COM (short username oryx)
I20260925 14:58:30.841724 14693 webserver.cc:460] Webserver started at http://127.8.13.195:45689/ using document root <none> and password file <none>
I20260925 14:58:30.841943 14693 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:30.841989 14693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:30.842120 14693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:30.843250 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "59ca4a67ba77482c9c2c363f12d3055e"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "e63a5903120d9aa187e14de62583f572"
server_key_iv: "4025207698fea691d0b782bc269abcff"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.843636 14693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "59ca4a67ba77482c9c2c363f12d3055e"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "e63a5903120d9aa187e14de62583f572"
server_key_iv: "4025207698fea691d0b782bc269abcff"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.844938 14693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:30.845726 14708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.845885 14693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:30.845964 14693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "59ca4a67ba77482c9c2c363f12d3055e"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "e63a5903120d9aa187e14de62583f572"
server_key_iv: "4025207698fea691d0b782bc269abcff"
server_key_version: "encryptionkey@0"
I20260925 14:58:30.846128 14693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:30.856890 14693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:30.858741 14693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:30.858863 14693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:30.859095 14693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:30.859385 14693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:30.859417 14693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.859440 14693 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:30.859452 14693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:30.865190 14693 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:39795
I20260925 14:58:30.865254 14821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:39795 every 8 connection(s)
I20260925 14:58:30.865551 14693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:30.872066 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14693
I20260925 14:58:30.872161 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:30.872366 8247 external_mini_cluster.cc:1428] Setting key cc1073293827b08badcb67cc0fa9df58
I20260925 14:58:30.874476 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.866239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:39671 (local address 127.8.13.254:39483)
0925 14:58:30.866301 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:30.866303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:30.866490 (+ 187us) server_negotiation.cc:407] Connection header received
0925 14:58:30.867051 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:30.867052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:30.867064 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:30.867116 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:30.868143 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.868643 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.869622 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.869825 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.869881 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.871919 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:30.871921 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:30.871921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:30.871929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:30.873218 (+ 1289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.873602 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.873602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.873603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.873656 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:30.873870 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:30.873870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:30.873871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:30.873982 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:30.874057 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:30.874307 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 14:58:30.874386 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:30.874652 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:30.866209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:39671)
0925 14:58:30.866373 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:30.866411 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:30.866906 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:30.867196 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:30.867200 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:30.867847 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:30.868052 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.868058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.868723 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.868724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.869540 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:30.869542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:30.869895 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:30.869896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:30.870018 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:30.870477 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:30.870501 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:30.871833 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:30.873304 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.873312 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.873314 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.873525 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.873712 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:30.873713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:30.873713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:30.873813 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:30.874037 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:30.874043 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:30.874188 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 14:58:30.874278 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 14:58:30.874439 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260925 14:58:30.875037 14822 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:30.875130 14822 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:30.875274 14822 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:30.875638 14363 ts_manager.cc:194] Registered new tserver with Master: 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:30.875991 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:39671
I20260925 14:58:30.884150 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.13.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.253.keytab.
Entry for principal oryx/127.8.13.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.253.keytab.
I20260925 14:58:30.910257 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:33593
--webserver_interface=127.8.13.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.253.keytab
--principal=oryx/127.8.13.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39483,127.8.13.253:33593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:30.989595 14827 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:30.989826 14827 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:30.989858 14827 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:30.991019 14827 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:30.991070 14827 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:30.991089 14827 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:30.991106 14827 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:30.991120 14827 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:30.992635 14827 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39483,127.8.13.253:33593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.253.keytab
--principal=oryx/127.8.13.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:33593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14827
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
W20260925 14:58:30.992756 14827 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.13.254:39483,127.8.13.253:33593'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260925 14:58:30.992913 14827 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:30.993122 14827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:30.997653 14833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.997685 14832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:30.997658 14835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:30.998073 14827 server_base.cc:1034] running on GCE node
I20260925 14:58:30.998268 14827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:30.998538 14827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:30.999708 14827 hybrid_clock.cc:648] HybridClock initialized: now 1790348310999692 us; error 29 us; skew 500 ppm
Sep 25 14:58:30 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.002151 14827 init.cc:377] Logged in from keytab as oryx/127.8.13.253@KRBTEST.COM (short username oryx)
I20260925 14:58:31.002624 14827 webserver.cc:460] Webserver started at http://127.8.13.253:45221/ using document root <none> and password file <none>
I20260925 14:58:31.002838 14827 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.002890 14827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.003026 14827 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:31.004153 14827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/instance:
uuid: "153aff0e44394f4a9f68cfb489c275ab"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "937b0bd191de0035a48605412f33673e"
server_key_iv: "6a474643fc24537dc104aa4614b74a93"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.004518 14827 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal/instance:
uuid: "153aff0e44394f4a9f68cfb489c275ab"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "937b0bd191de0035a48605412f33673e"
server_key_iv: "6a474643fc24537dc104aa4614b74a93"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.005908 14827 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:31.006712 14842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.006928 14827 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:31.007018 14827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "153aff0e44394f4a9f68cfb489c275ab"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "937b0bd191de0035a48605412f33673e"
server_key_iv: "6a474643fc24537dc104aa4614b74a93"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.007082 14827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.020690 14827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.022559 14827 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:31.022711 14827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.026449 14827 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.253:33593
I20260925 14:58:31.026499 14894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.253:33593 every 8 connection(s)
I20260925 14:58:31.026827 14827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
I20260925 14:58:31.027805 14895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:31.029379 14895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.253@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:31.034857 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 14827
I20260925 14:58:31.037945 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.030029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50730 (local address 127.8.13.254:39483)
0925 14:58:31.030102 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.030104 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.030288 (+ 184us) server_negotiation.cc:407] Connection header received
0925 14:58:31.030741 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.030742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.030751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.030809 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.031418 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.031944 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.032887 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.033061 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.033117 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.035308 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.035310 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.035310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.035317 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.036664 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.037030 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.037031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.037031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.037090 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.037306 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.037306 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.037307 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.037452 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.037509 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.037772 (+ 263us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.037855 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:31.038116 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.030008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50730)
0925 14:58:31.030174 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.030217 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.030658 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.030874 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.030878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.031149 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.031352 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.031356 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.032007 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.032008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.032823 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.032825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.033128 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.033129 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.033245 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.033735 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.033745 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.035212 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.036747 (+ 1535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.036754 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.036756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.036963 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.037143 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.037144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.037145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.037250 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.037510 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.037516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.037657 (+ 141us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.037744 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.037900 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260925 14:58:31.038669 14895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.253@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
I20260925 14:58:31.043612 14899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.035447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36582 (local address 127.8.13.253:33593)
0925 14:58:31.035616 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.035621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.035901 (+ 280us) server_negotiation.cc:407] Connection header received
0925 14:58:31.035927 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.035930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.035945 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.036007 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.036532 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.037222 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.037920 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.038339 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.038403 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.040806 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.040815 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.040815 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.040822 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.042364 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.042703 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.042706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.042707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.042762 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.042938 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.042940 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.042940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.043113 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.043278 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.043324 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.043405 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260925 14:58:31.045451 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.038947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36594)
0925 14:58:31.038982 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.038992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.039044 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.039250 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.039251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.039410 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.039464 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.039465 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.040308 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.040309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.040836 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.040838 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.041192 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.041193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.041255 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.041552 (+ 297us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.041555 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.042653 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.044754 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.044755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.044755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.044902 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.045036 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.045036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.045037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.045078 (+ 41us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.045229 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.045230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.045261 (+ 31us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.045304 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.045372 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260925 14:58:31.045535 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.038993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36594 (local address 127.8.13.253:33593)
0925 14:58:31.039151 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.039152 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.039159 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:31.039173 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.039174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.039183 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.039241 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.039557 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.040233 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.040905 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.041126 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.041182 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.042730 (+ 1548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.042731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.042731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.042737 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.044682 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.044956 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.044957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.044957 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.045003 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.045112 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.045113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.045113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.045197 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.045238 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.045319 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.045414 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260925 14:58:31.046679 14895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Bootstrap starting.
I20260925 14:58:31.047235 14895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:31.047449 14895 log.cc:824] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:31.048228 14895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: No bootstrap required, opened a new log
I20260925 14:58:31.049708 14895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } }
I20260925 14:58:31.049870 14895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:31.049901 14895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 153aff0e44394f4a9f68cfb489c275ab, State: Initialized, Role: FOLLOWER
I20260925 14:58:31.050040 14895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } }
I20260925 14:58:31.050350 14895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:31.050330 14902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } } }
I20260925 14:58:31.050486 14902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:58:31.051859 14913 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:31.051918 14913 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:31.053581 14827 master_runner.cc:428] Detected that this master 153aff0e44394f4a9f68cfb489c275ab is joining an existing cluster
I20260925 14:58:31.053633 14827 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.13.253:33593
I20260925 14:58:31.061002 14919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.054737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36604)
0925 14:58:31.054872 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.054881 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.054946 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.055107 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.055108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.055305 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.055369 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.055371 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.056299 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.056300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.057191 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.057193 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.057620 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.057666 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.058030 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.058034 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.058420 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.060020 (+ 1600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.060021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.060184 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.060344 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.060345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.060399 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.060628 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.060628 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.060671 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.060752 (+ 81us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.060859 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":31,"thread_start_us":73,"threads_started":1}
I20260925 14:58:31.061079 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.054667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36604 (local address 127.8.13.253:33593)
0925 14:58:31.054703 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.054704 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.054927 (+ 223us) server_negotiation.cc:407] Connection header received
0925 14:58:31.055014 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.055014 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.055022 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.055063 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.055437 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.056216 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057270 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.057529 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057584 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.058504 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.058505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.058505 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.058511 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.059952 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060238 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.060240 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.060240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.060286 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060457 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.060458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.060459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.060576 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.060638 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.060802 (+ 164us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.060927 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":18}
I20260925 14:58:31.061929 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.054641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50738 (local address 127.8.13.254:39483)
0925 14:58:31.054686 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.054688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.054932 (+ 244us) server_negotiation.cc:407] Connection header received
0925 14:58:31.054987 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.054988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.055000 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.055046 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.055498 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.056269 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057003 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.057216 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057268 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.058510 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.058512 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.058512 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.058520 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.060378 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060922 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.060923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.060924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.060982 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.061162 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.061164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.061164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.061302 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.061369 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.061753 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.061835 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:31.062256 14920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.054591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50738)
0925 14:58:31.054865 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.054875 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.054928 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.055060 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.055061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.055332 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.055394 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.055395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.056342 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.056343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.056927 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.056929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.057333 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.057334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.057405 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.057889 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.057893 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.058420 (+ 527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.060454 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.060455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.060456 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.060844 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.061051 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.061051 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.061052 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.061108 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.061579 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.061580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.061623 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.062047 (+ 424us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.062152 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":99,"threads_started":1}
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.254@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
I20260925 14:58:31.071339 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.064361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36616 (local address 127.8.13.253:33593)
0925 14:58:31.064396 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.064398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.064546 (+ 148us) server_negotiation.cc:407] Connection header received
0925 14:58:31.064640 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.064641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.064651 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.064714 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.065129 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.065592 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.066370 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.066660 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.066700 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.069026 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.069028 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.069028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.069035 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.070259 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.070605 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.070607 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.070607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.070658 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.070818 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.070819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.070819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.070920 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.070975 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.071164 (+ 189us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.071239 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 14:58:31.071342 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.064317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36616)
0925 14:58:31.064453 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.064493 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.064563 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.064725 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.064730 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.065001 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.065054 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.065056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.065663 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.065664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.066273 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.066274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.066776 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.066777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.066860 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.067386 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.067391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.068942 (+ 1551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.070339 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.070347 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.070349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.070542 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.070711 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.070711 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.070712 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.070770 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.070972 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.070981 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.071036 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.071112 (+ 76us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.071242 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 14:58:31.072098 14363 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.13.253:33593: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab"
member_type: NON_VOTER
last_known_addr {
host: "127.8.13.253"
port: 33593
}
attrs {
promote: true
}
}
dest_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16"
cas_config_opid_index: -1
I20260925 14:58:31.072408 14363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } }
I20260925 14:58:31.072854 14923 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } }
I20260925 14:58:31.073169 14869 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:31.073711 14923 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.13.253:33593
I20260925 14:58:31.073930 14827 master.cc:537] Master@127.8.13.253:33593 shutting down...
I20260925 14:58:31.073908 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } }
I20260925 14:58:31.074116 14869 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Refusing update from remote peer 3577c8dc9c9a4d8bbd77055ee1786b16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260925 14:58:31.073977 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:31.074535 14923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 14:58:31.075819 14350 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260925 14:58:31.076229 14350 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 -> Peer 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253:33593): Couldn't send request to peer 153aff0e44394f4a9f68cfb489c275ab. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:58:31.076584 14827 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:31.076675 14827 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:31.076725 14827 tablet_replica.cc:330] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: stopping tablet replica
I20260925 14:58:31.089588 14827 master.cc:552] Master@127.8.13.253:33593 shutdown complete.
I20260925 14:58:31.089643 14827 master_runner.cc:305] Clearing existing system tablet
I20260925 14:58:31.089766 14827 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.089810 14827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.090400 14827 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.001s
I20260925 14:58:31.090823 14927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.090960 14827 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.091040 14827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.091068 14827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "153aff0e44394f4a9f68cfb489c275ab"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "937b0bd191de0035a48605412f33673e"
server_key_iv: "6a474643fc24537dc104aa4614b74a93"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.091910 14827 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:31.093693 14827 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:58:31.093768 14827 log.cc:1196] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260925 14:58:31.093959 14827 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Deleting consensus metadata
I20260925 14:58:31.094072 14827 master_runner.cc:315] Copying system tablet from 127.8.13.254:39483
I20260925 14:58:31.094682 14827 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: tablet copy: Beginning tablet copy session from remote peer at address 127.8.13.254:39483
I20260925 14:58:31.100832 14933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.095008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50754)
0925 14:58:31.095117 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.095145 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.095204 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.095339 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.095340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.095500 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.095613 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.095615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.096532 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.096532 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.097097 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.097099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.097560 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.097561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.097611 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.098044 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.098049 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.098355 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.099910 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.099911 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.099911 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.100074 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.100256 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.100256 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.100256 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.100303 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.100575 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.100575 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.100616 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.100664 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.100748 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":72,"threads_started":1}
I20260925 14:58:31.100862 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.095084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50754 (local address 127.8.13.254:39483)
0925 14:58:31.095132 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.095134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.095207 (+ 73us) server_negotiation.cc:407] Connection header received
0925 14:58:31.095243 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.095244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.095255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.095334 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.095684 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.096459 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.097175 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.097481 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.097565 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.098430 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.098432 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.098432 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.098438 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.099837 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.100139 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.100140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.100140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.100200 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.100353 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.100354 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.100354 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.100513 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.100587 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.100689 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.100765 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:58:31.101487 14383 tablet_copy_service.cc:140] P 3577c8dc9c9a4d8bbd77055ee1786b16: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 153aff0e44394f4a9f68cfb489c275ab ({username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:50754)
I20260925 14:58:31.101603 14383 tablet_copy_service.cc:161] P 3577c8dc9c9a4d8bbd77055ee1786b16: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 153aff0e44394f4a9f68cfb489c275ab at {username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:50754: session id = 153aff0e44394f4a9f68cfb489c275ab-00000000000000000000000000000000
I20260925 14:58:31.102391 14383 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:58:31.103041 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:58:31.104416 14827 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: tablet copy: Starting download of 0 data blocks...
I20260925 14:58:31.104578 14827 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: tablet copy: Starting download of 1 WAL segments...
I20260925 14:58:31.105444 14827 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:58:31.106418 14383 tablet_copy_service.cc:342] P 3577c8dc9c9a4d8bbd77055ee1786b16: Request end of tablet copy session 153aff0e44394f4a9f68cfb489c275ab-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:50754
I20260925 14:58:31.106473 14383 tablet_copy_service.cc:433] P 3577c8dc9c9a4d8bbd77055ee1786b16: ending tablet copy session 153aff0e44394f4a9f68cfb489c275ab-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 153aff0e44394f4a9f68cfb489c275ab
I20260925 14:58:31.108843 14827 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.109866 14934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.110001 14937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.110139 14935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.110239 14827 server_base.cc:1034] running on GCE node
I20260925 14:58:31.110320 14827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.110443 14827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.111608 14827 hybrid_clock.cc:648] HybridClock initialized: now 1790348311111595 us; error 26 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.112887 14827 init.cc:377] Logged in from keytab as oryx/127.8.13.253@KRBTEST.COM (short username oryx)
I20260925 14:58:31.113138 14827 webserver.cc:460] Webserver started at http://127.8.13.253:38615/ using document root <none> and password file <none>
I20260925 14:58:31.113255 14827 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.113283 14827 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.113852 14827 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:31.114332 14944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.114459 14827 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.114501 14827 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "153aff0e44394f4a9f68cfb489c275ab"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "937b0bd191de0035a48605412f33673e"
server_key_iv: "6a474643fc24537dc104aa4614b74a93"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.114560 14827 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.122386 14827 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.123970 14827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.126827 14827 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.253:33593
I20260925 14:58:31.126840 14996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.253:33593 every 8 connection(s)
I20260925 14:58:31.127166 14827 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
I20260925 14:58:31.127955 14997 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:31.128393 14997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Bootstrap starting.
I20260925 14:58:31.130503 14997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:31.130672 14997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Bootstrap complete.
I20260925 14:58:31.130848 14997 raft_consensus.cc:348] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } }
I20260925 14:58:31.130918 14997 raft_consensus.cc:729] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 153aff0e44394f4a9f68cfb489c275ab, State: Initialized, Role: LEARNER
I20260925 14:58:31.130941 14997 consensus_queue.cc:260] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } }
I20260925 14:58:31.131107 14997 sys_catalog.cc:564] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:31.131096 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } }
I20260925 14:58:31.131220 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: LEARNER
I20260925 14:58:31.132404 15011 catalog_manager.cc:1259] Loaded cluster ID: 9d73cb0257cd4c7d84a0432090fe0440
I20260925 14:58:31.132452 15011 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: loading cluster ID for follower catalog manager: success
I20260925 14:58:31.133409 15011 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: acquiring CA information for follower catalog manager: success
I20260925 14:58:31.133674 15011 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:58:31.151304 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.145288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36634 (local address 127.8.13.253:33593)
0925 14:58:31.145420 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.145422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.145428 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:58:31.145450 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.145450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.145459 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.145520 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.146062 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.146725 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.147435 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.147673 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.147734 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.148905 (+ 1171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.148907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.148907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.148915 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.150282 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.150584 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.150585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.150586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.150653 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.150816 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.150816 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.150817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.150923 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.151009 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.151109 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.151189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
WARNING: no policy specified for oryx/127.8.13.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.252@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.252.keytab.
Entry for principal oryx/127.8.13.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.252.keytab.
I20260925 14:58:31.178545 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:33595
--webserver_interface=127.8.13.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.252.keytab
--principal=oryx/127.8.13.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:31.266013 15015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:31.266243 15015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:31.266266 15015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:31.267423 15015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:31.267462 15015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:31.267475 15015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:31.267486 15015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:31.267503 15015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:31.269101 15015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.252.keytab
--principal=oryx/127.8.13.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:33595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:31.269325 15015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:31.269505 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.274246 15021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.274241 15020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.274246 15023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.274432 15015 server_base.cc:1034] running on GCE node
I20260925 14:58:31.274644 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.274930 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.276100 15015 hybrid_clock.cc:648] HybridClock initialized: now 1790348311276041 us; error 51 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.278579 15015 init.cc:377] Logged in from keytab as oryx/127.8.13.252@KRBTEST.COM (short username oryx)
I20260925 14:58:31.279038 15015 webserver.cc:460] Webserver started at http://127.8.13.252:40527/ using document root <none> and password file <none>
I20260925 14:58:31.279286 15015 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.279332 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.279453 15015 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:31.280519 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/instance:
uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
format_stamp: "Formatted at 2026-09-25 14:58:31 on dist-test-slave-15hz"
server_key: "4e913c41ddb53df14378588ec32a9621"
server_key_iv: "3504f2ba6e65f31fd43c7e8c8ae7dae5"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.280850 15015 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal/instance:
uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
format_stamp: "Formatted at 2026-09-25 14:58:31 on dist-test-slave-15hz"
server_key: "4e913c41ddb53df14378588ec32a9621"
server_key_iv: "3504f2ba6e65f31fd43c7e8c8ae7dae5"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.282097 15015 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:31.282753 15030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.282904 15015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:31.282980 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
format_stamp: "Formatted at 2026-09-25 14:58:31 on dist-test-slave-15hz"
server_key: "4e913c41ddb53df14378588ec32a9621"
server_key_iv: "3504f2ba6e65f31fd43c7e8c8ae7dae5"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.283044 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.297537 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.299474 15015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:31.299646 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.303501 15015 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.252:33595
I20260925 14:58:31.303550 15082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.252:33595 every 8 connection(s)
I20260925 14:58:31.303902 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
I20260925 14:58:31.304844 15083 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:31.306358 15083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.252@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:31.313912 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15015
I20260925 14:58:31.315131 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.307184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50760 (local address 127.8.13.254:39483)
0925 14:58:31.307260 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.307263 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.307357 (+ 94us) server_negotiation.cc:407] Connection header received
0925 14:58:31.307882 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.307883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.307892 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.307942 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.308506 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.308995 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.310040 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.310226 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.310270 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.312389 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.312390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.312391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.312397 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.313529 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.313901 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.313902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.313903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.313966 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.314227 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.314228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.314228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.314364 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.314426 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.314916 (+ 490us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.314999 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 14:58:31.315229 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.307002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50760)
0925 14:58:31.307228 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.307269 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.307798 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.308000 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.308003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.308244 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.308440 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.308444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.309093 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.309094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.309969 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.309972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.310283 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.310284 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.310395 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.310900 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.310908 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.312296 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.313595 (+ 1299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.313601 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.313603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.313819 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.314043 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.314043 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.314044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.314165 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.314427 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.314435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.314769 (+ 334us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.314871 (+ 102us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.315020 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":102,"threads_started":1}
I20260925 14:58:31.315851 15083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.252@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
I20260925 14:58:31.322934 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.316176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36644 (local address 127.8.13.253:33593)
0925 14:58:31.316235 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.316237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.316244 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:31.316269 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.316270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.316283 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.316326 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.316581 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.317048 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.317822 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.318031 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.318082 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.319874 (+ 1792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.319875 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.319876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.319885 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.321652 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.322073 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.322074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.322075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.322137 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.322323 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.322324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.322324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.322468 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.322536 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.322729 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.322830 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 14:58:31.323244 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.316108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36644)
0925 14:58:31.316147 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.316157 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.316192 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.316333 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.316334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.316458 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.316514 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.316516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.317138 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.317139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.317751 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.317754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.318096 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.318097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.318160 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.318510 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.318513 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.319708 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.321748 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.321749 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.321749 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.321993 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.322201 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.322201 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.322202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.322264 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.322540 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.322541 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.322597 (+ 56us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.323023 (+ 426us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.323141 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":19}
I20260925 14:58:31.323668 15083 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } has no permanent_uuid. Determining permanent_uuid...
I20260925 14:58:31.324113 15086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.314485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.8.13.252:33595)
0925 14:58:31.314661 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.314666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.314684 (+ 18us) server_negotiation.cc:407] Connection header received
0925 14:58:31.314704 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.314706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.314719 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.314790 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.315436 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.316404 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.317154 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.317472 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.317521 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.320469 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.320482 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.320482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.320489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.322564 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.322964 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.322967 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.322968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.323029 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.323252 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.323253 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.323253 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.323433 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.323925 (+ 492us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.323971 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.324036 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.252@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
I20260925 14:58:31.330106 15085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.323907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35852)
0925 14:58:31.324069 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.324079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.324120 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.324407 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.324408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.324528 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.324556 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.324557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.325170 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.325170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.325654 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.325656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.325989 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.325989 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.326035 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.326338 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.326341 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.327440 (+ 1099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.329240 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.329241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.329241 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.329389 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.329546 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.329547 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.329547 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.329597 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.329807 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.329808 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.329844 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.329904 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.330000 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260925 14:58:31.330175 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.323955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35852 (local address 127.8.13.252:33595)
0925 14:58:31.324094 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.324096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.324107 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:31.324334 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.324335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.324343 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.324399 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.324612 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.325099 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.325717 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.325934 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.325987 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.327510 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.327511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.327512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.327518 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.329166 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.329444 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.329445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.329446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.329497 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.329653 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.329653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.329654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.329758 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.329811 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.329928 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.330048 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 14:58:31.331617 15083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Bootstrap starting.
I20260925 14:58:31.332228 15083 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:31.332500 15083 log.cc:824] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:31.333209 15083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: No bootstrap required, opened a new log
I20260925 14:58:31.334568 15083 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } }
I20260925 14:58:31.334697 15083 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:31.334719 15083 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4576b5c4e5d4f8c98c4327ca7fc1dd0, State: Initialized, Role: FOLLOWER
I20260925 14:58:31.334820 15083 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } }
I20260925 14:58:31.335038 15089 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } } }
I20260925 14:58:31.335093 15083 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:31.335140 15089 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 14:58:31.336680 15100 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:31.336730 15100 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:31.338948 15015 master_runner.cc:428] Detected that this master e4576b5c4e5d4f8c98c4327ca7fc1dd0 is joining an existing cluster
I20260925 14:58:31.338987 15015 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.13.252:33595
I20260925 14:58:31.340265 15089 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:31.340351 15089 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } }
I20260925 14:58:31.340581 15089 leader_election.cc:288] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253:33593)
I20260925 14:58:31.340756 14971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "153aff0e44394f4a9f68cfb489c275ab" is_pre_election: true
I20260925 14:58:31.340863 14971 raft_consensus.cc:1799] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Handling vote request from an unknown peer e4576b5c4e5d4f8c98c4327ca7fc1dd0
I20260925 14:58:31.340822 14373 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" is_pre_election: true
I20260925 14:58:31.340905 14971 raft_consensus.cc:2396] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate e4576b5c4e5d4f8c98c4327ca7fc1dd0 for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260925 14:58:31.341193 15034 leader_election.cc:302] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters: 153aff0e44394f4a9f68cfb489c275ab, 3577c8dc9c9a4d8bbd77055ee1786b16
I20260925 14:58:31.341313 15089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:31.342267 15089 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 14:58:31.365914 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.340018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.8.13.254:39483)
0925 14:58:31.340132 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.340134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.340253 (+ 119us) server_negotiation.cc:407] Connection header received
0925 14:58:31.340339 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.340341 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.340351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.340392 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.341212 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.341787 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.342684 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.342883 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.342933 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.343886 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.343888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.343888 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.343896 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.345281 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.364222 (+ 18941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.364224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.364225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.364294 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.364865 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.364867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.364867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.365034 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.365110 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.365713 (+ 603us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.365811 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260925 14:58:31.366567 15107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.339927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50772)
0925 14:58:31.340189 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.340200 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.340278 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.340406 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.340407 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.340616 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.340683 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.340685 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.341861 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.341861 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.342606 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.342609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.342958 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.342959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.343022 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.343467 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.343472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.343813 (+ 341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.345344 (+ 1531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.345345 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.345345 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.364119 (+ 18774us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.364637 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.364638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.364639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.364718 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.365114 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.365115 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.365371 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.366320 (+ 949us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.366457 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":95,"threads_started":1}
I20260925 14:58:31.367400 15106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.340201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35866)
0925 14:58:31.347689 (+ 7488us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.347703 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.347767 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.347890 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.347890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.348102 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.348166 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.348168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.348826 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.348827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.349371 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.349374 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.349750 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.349751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.349806 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.350186 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.350190 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.364496 (+ 14306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.366368 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366369 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.366369 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.366563 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.366743 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366743 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.366743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.366801 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.367110 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.367110 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.367144 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.367206 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.367298 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7463,"thread_start_us":544,"threads_started":1}
I20260925 14:58:31.367486 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.340085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36658 (local address 127.8.13.253:33593)
0925 14:58:31.340132 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.340134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.340866 (+ 732us) server_negotiation.cc:407] Connection header received
0925 14:58:31.340953 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.340954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.340964 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.340992 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.342492 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.343271 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.344812 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.345121 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.345199 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.364367 (+ 19168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.364370 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.364370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.364379 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.365964 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366467 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.366469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.366469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.366524 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366715 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.366716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.366716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.366836 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.366875 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.367318 (+ 443us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.367404 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:31.367795 15108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.339931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36658)
0925 14:58:31.340808 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.340818 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.340889 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.341053 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.341054 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.341243 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.342396 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.342399 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.343368 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.343369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.344660 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.344663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.345206 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.345207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.345277 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.345597 (+ 320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.345628 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.364245 (+ 18617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.366049 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366050 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.366050 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.366379 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.366591 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.366592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.366645 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.366919 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.366920 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.366963 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.367593 (+ 630us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.367691 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"spinlock_wait_cycles":10496,"thread_start_us":98,"threads_started":1}
I20260925 14:58:31.368801 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.340200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35866 (local address 127.8.13.252:33595)
0925 14:58:31.340258 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.340259 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.347766 (+ 7507us) server_negotiation.cc:407] Connection header received
0925 14:58:31.347823 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.347824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.347832 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.347879 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.348235 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.348761 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.349440 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.349693 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.349747 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.364582 (+ 14835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.364583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.364583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.364590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.366286 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366640 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.366641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.366641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.366685 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.366855 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.366856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.366856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.367033 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.367072 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.368656 (+ 1584us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.368732 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348310, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.254@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
I20260925 14:58:31.375306 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14425
I20260925 14:58:31.380167 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:40377
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:39483
--webserver_port=33453
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:31.380403 15087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.369115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35872 (local address 127.8.13.252:33595)
0925 14:58:31.369233 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.369234 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.369240 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:58:31.369259 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.369259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.369267 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.369308 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.369604 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.370045 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.371027 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.371314 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.371385 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.373361 (+ 1976us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.373362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.373363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.373369 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.374533 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.374854 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.374855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.374855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.374912 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.375113 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.375114 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.375114 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.375311 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.375406 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.380203 (+ 4797us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.380323 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260925 14:58:31.380496 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.368922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35872)
0925 14:58:31.369005 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.369014 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.369048 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.369331 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.369332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.369493 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.369541 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.369543 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.370109 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.370110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.370944 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.370948 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.371389 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.371390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.371456 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.371849 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.371854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.373287 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.374611 (+ 1324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.374612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.374613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.374798 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.374969 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.374971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.374971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.375044 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.375375 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.375376 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.380054 (+ 4678us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.380149 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.380307 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260925 14:58:31.381496 14363 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.13.252:33595: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
member_type: NON_VOTER
last_known_addr {
host: "127.8.13.252"
port: 33595
}
attrs {
promote: true
}
}
dest_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16"
cas_config_opid_index: 5
I20260925 14:58:31.381749 14363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } }
I20260925 14:58:31.382112 14922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.382880 14922 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.13.252:33595
I20260925 14:58:31.382951 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.383014 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:31.383145 15015 master.cc:537] Master@127.8.13.252:33595 shutting down...
W20260925 14:58:31.383737 14349 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 -> Peer e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595): Couldn't send request to peer e4576b5c4e5d4f8c98c4327ca7fc1dd0. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260925 14:58:31.386298 15015 raft_consensus.cc:2229] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:31.386389 15015 raft_consensus.cc:2258] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:31.386431 15015 tablet_replica.cc:330] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: stopping tablet replica
I20260925 14:58:31.388124 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.383198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36668)
0925 14:58:31.383631 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.383643 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.383687 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.384453 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.384454 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.384456 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:31.384503 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.384505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.385456 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.385457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.386754 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.386758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.387834 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.387836 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.387911 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.387945 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.388001 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.388050 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414}
I20260925 14:58:31.388381 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.383260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36668 (local address 127.8.13.253:33593)
0925 14:58:31.384259 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.384262 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.384272 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:31.384298 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.384299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.384324 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.384388 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:31.384567 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.385335 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.386857 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.387475 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.387590 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.387681 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.388271 (+ 590us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.388318 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":971}
I20260925 14:58:31.388684 14971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Refusing update from remote peer 3577c8dc9c9a4d8bbd77055ee1786b16: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260925 14:58:31.388881 14923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260925 14:58:31.389483 14971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.389796 15015 master.cc:552] Master@127.8.13.252:33595 shutdown complete.
I20260925 14:58:31.389832 15015 master_runner.cc:305] Clearing existing system tablet
I20260925 14:58:31.389953 15015 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.389988 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.390637 15015 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:31.390872 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3577c8dc9c9a4d8bbd77055ee1786b16. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.390964 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: LEARNER
I20260925 14:58:31.390998 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: NON_VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: true } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.391036 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: LEARNER
I20260925 14:58:31.391233 14922 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: attempting to promote NON_VOTER 153aff0e44394f4a9f68cfb489c275ab to VOTER
I20260925 14:58:31.391278 15113 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.391467 15015 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.391469 14922 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } }
I20260925 14:58:31.391515 15015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:31.391748 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
format_stamp: "Formatted at 2026-09-25 14:58:31 on dist-test-slave-15hz"
server_key: "4e913c41ddb53df14378588ec32a9621"
server_key_iv: "3504f2ba6e65f31fd43c7e8c8ae7dae5"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.392243 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.392081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35888)
0925 14:58:31.392168 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.392212 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:33595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260925 14:58:31.392343 14971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 LEARNER]: Refusing update from remote peer 3577c8dc9c9a4d8bbd77055ee1786b16: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260925 14:58:31.392508 14923 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260925 14:58:31.392560 15015 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:31.393061 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.392980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35900)
0925 14:58:31.393013 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.393036 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:33595: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 14:58:31.393113 14349 consensus_peers.cc:487] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 -> Peer e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595): Couldn't send request to peer e4576b5c4e5d4f8c98c4327ca7fc1dd0. Status: Network error: Client connection negotiation failed: client connection to 127.8.13.252:33595: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:58:31.393414 14922 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.394276 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.394358 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:31.394433 14922 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.394481 14922 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:31.394857 14971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.395851 15015 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260925 14:58:31.395813 15000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.395890 15000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:31.395915 15015 log.cc:1196] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260925 14:58:31.396104 15015 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Deleting consensus metadata
I20260925 14:58:31.396204 15015 master_runner.cc:315] Copying system tablet from 127.8.13.254:39483
I20260925 14:58:31.396793 15015 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: tablet copy: Beginning tablet copy session from remote peer at address 127.8.13.254:39483
I20260925 14:58:31.403735 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.397212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50784 (local address 127.8.13.254:39483)
0925 14:58:31.397249 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.397251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.397260 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:31.397315 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.397315 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.397325 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.397381 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.397723 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.398466 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.399512 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.399820 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.399893 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.401009 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.401010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.401010 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.401018 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.402560 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.402946 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.402948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.402948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.403010 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.403188 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.403188 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.403189 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.403320 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.403384 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.403520 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.403642 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:31.403741 15120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.397049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:50784)
0925 14:58:31.397194 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.397205 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.397263 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.397385 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.397386 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.397587 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.397651 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.397654 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.398546 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.398547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.399421 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.399423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.399898 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.399899 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.399969 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.400481 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.400487 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.400912 (+ 425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.402641 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.402643 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.402643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.402871 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.403072 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.403073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.403073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.403133 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.403378 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.403379 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.403427 (+ 48us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.403489 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.403631 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260925 14:58:31.404368 14383 tablet_copy_service.cc:140] P 3577c8dc9c9a4d8bbd77055ee1786b16: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e4576b5c4e5d4f8c98c4327ca7fc1dd0 ({username='oryx', principal='oryx/127.8.13.252@KRBTEST.COM'} at 127.0.0.1:50784)
I20260925 14:58:31.404431 14383 tablet_copy_service.cc:161] P 3577c8dc9c9a4d8bbd77055ee1786b16: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e4576b5c4e5d4f8c98c4327ca7fc1dd0 at {username='oryx', principal='oryx/127.8.13.252@KRBTEST.COM'} at 127.0.0.1:50784: session id = e4576b5c4e5d4f8c98c4327ca7fc1dd0-00000000000000000000000000000000
I20260925 14:58:31.405047 14383 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Tablet Copy: opened 0 blocks and 1 log segments
I20260925 14:58:31.405588 15015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260925 14:58:31.406905 15015 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: tablet copy: Starting download of 0 data blocks...
I20260925 14:58:31.407048 15015 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: tablet copy: Starting download of 1 WAL segments...
I20260925 14:58:31.407904 15015 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260925 14:58:31.408946 14383 tablet_copy_service.cc:342] P 3577c8dc9c9a4d8bbd77055ee1786b16: Request end of tablet copy session e4576b5c4e5d4f8c98c4327ca7fc1dd0-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.13.252@KRBTEST.COM'} at 127.0.0.1:50784
I20260925 14:58:31.409003 14383 tablet_copy_service.cc:433] P 3577c8dc9c9a4d8bbd77055ee1786b16: ending tablet copy session e4576b5c4e5d4f8c98c4327ca7fc1dd0-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e4576b5c4e5d4f8c98c4327ca7fc1dd0
I20260925 14:58:31.411478 15015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.412652 15121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.412799 15124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.412839 15122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.415738 15015 server_base.cc:1034] running on GCE node
I20260925 14:58:31.415848 15015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.415979 15015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.417097 15015 hybrid_clock.cc:648] HybridClock initialized: now 1790348311417090 us; error 25 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.418627 15015 init.cc:377] Logged in from keytab as oryx/127.8.13.252@KRBTEST.COM (short username oryx)
I20260925 14:58:31.418983 15015 webserver.cc:460] Webserver started at http://127.8.13.252:36051/ using document root <none> and password file <none>
I20260925 14:58:31.419101 15015 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.419134 15015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.419860 15015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:31.420431 15131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.420603 15015 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:31.420653 15015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
format_stamp: "Formatted at 2026-09-25 14:58:31 on dist-test-slave-15hz"
server_key: "4e913c41ddb53df14378588ec32a9621"
server_key_iv: "3504f2ba6e65f31fd43c7e8c8ae7dae5"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.420712 15015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.431401 15015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.433349 15015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.436206 15015 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.252:33595
I20260925 14:58:31.436568 15015 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
I20260925 14:58:31.437444 15184 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:31.437939 15184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Bootstrap starting.
I20260925 14:58:31.439199 15183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.252:33595 every 8 connection(s)
I20260925 14:58:31.440819 15184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:31.441018 15184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: Bootstrap complete.
I20260925 14:58:31.441207 15184 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } }
I20260925 14:58:31.441295 15184 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e4576b5c4e5d4f8c98c4327ca7fc1dd0, State: Initialized, Role: LEARNER
I20260925 14:58:31.441331 15184 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } }
I20260925 14:58:31.441514 15184 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:31.441517 15187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.441754 15187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: LEARNER
I20260925 14:58:31.443099 15198 catalog_manager.cc:1259] Loaded cluster ID: 9d73cb0257cd4c7d84a0432090fe0440
I20260925 14:58:31.443186 15198 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: loading cluster ID for follower catalog manager: success
I20260925 14:58:31.444413 15198 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: acquiring CA information for follower catalog manager: success
I20260925 14:58:31.444818 15198 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260925 14:58:31.480273 15109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:31.480466 15109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:31.480489 15109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:31.481612 15109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:31.481647 15109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:31.481671 15109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:31.483078 15109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:40377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=33453
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15109
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:31.483258 15109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:31.483434 15109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.488227 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.488384 15109 server_base.cc:1034] running on GCE node
W20260925 14:58:31.488248 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.488255 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.488654 15109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.488883 15109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.490024 15109 hybrid_clock.cc:648] HybridClock initialized: now 1790348311490003 us; error 36 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.492380 15109 init.cc:377] Logged in from keytab as oryx/127.8.13.193@KRBTEST.COM (short username oryx)
I20260925 14:58:31.492754 15109 webserver.cc:460] Webserver started at http://127.8.13.193:33453/ using document root <none> and password file <none>
I20260925 14:58:31.492923 15109 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.492972 15109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.494206 15109 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:31.494791 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.494968 15109 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:31.495051 15109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "71b2cc4469ea000a06d682c7df3a987f"
server_key_iv: "6e29ee0cccb916864ce9a06326e9b4b9"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.495375 15109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.510551 15109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.512799 15109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:31.512914 15109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.513111 15109 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:31.513357 15109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:31.513386 15109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.513407 15109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:31.513422 15109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.518944 15109 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:40377
I20260925 14:58:31.519016 15326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:40377 every 8 connection(s)
I20260925 14:58:31.519327 15109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
I20260925 14:58:31.528368 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15109
I20260925 14:58:31.528532 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14559
I20260925 14:58:31.529515 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:42335 (local address 127.8.13.254:39483)
0925 14:58:31.520607 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.520609 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.520617 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:31.521000 (+ 383us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.521001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.521012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.521059 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.521877 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522661 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.523922 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.524184 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524242 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.526298 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.526299 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.526300 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.526307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.527901 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528389 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.528391 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.528391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.528456 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528824 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.528825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.528825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.528962 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.529000 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.529346 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.529425 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:58:31.529598 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:39571 (local address 127.8.13.253:33593)
0925 14:58:31.520094 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.520096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.520344 (+ 248us) server_negotiation.cc:407] Connection header received
0925 14:58:31.520927 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.520928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.520942 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.520986 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.521899 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522664 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.523762 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.523999 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524062 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.527080 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.527082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.527082 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.527090 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.528556 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528874 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.528875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.528876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.528921 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.529087 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.529088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.529088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.529214 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.529252 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.529426 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.529499 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:31.530104 15331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.8.13.193:39571)
0925 14:58:31.520222 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.520266 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.520830 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.521068 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.521072 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.521538 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.521783 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.521789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522734 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.522735 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.523681 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.523683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.524070 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.524186 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.524706 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.524715 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.527007 (+ 2292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.528651 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.528653 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.528814 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.528981 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528981 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.528982 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.529029 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.529291 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.529292 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.529330 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.529687 (+ 357us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.529860 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":104,"thread_start_us":79,"threads_started":1}
I20260925 14:58:31.530180 15334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:42335)
0925 14:58:31.520546 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.520555 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.520839 (+ 284us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.521078 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.521079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.521582 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.521781 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.521786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522857 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.522858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.523839 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.523842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.524252 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.524305 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.524818 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.524823 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.526208 (+ 1385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.527989 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.527998 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.528001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.528301 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.528611 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.528611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.528612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.528751 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.529040 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.529046 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.529205 (+ 159us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.529974 (+ 769us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.530070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":10,"thread_start_us":58,"threads_started":1}
I20260925 14:58:31.532884 15328 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:31.532991 15328 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.533187 15328 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:31.533404 15329 heartbeater.cc:347] Connected to a master server at 127.8.13.253:33593
I20260925 14:58:31.533461 15329 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.533529 15329 heartbeater.cc:511] Master 127.8.13.253:33593 requested a full tablet report, sending...
I20260925 14:58:31.533783 14363 ts_manager.cc:194] Re-registered known tserver with Master: ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377)
I20260925 14:58:31.533903 14961 ts_manager.cc:194] Registered new tserver with Master: ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377)
I20260925 14:58:31.534210 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:42335
I20260925 14:58:31.534587 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:45079
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:39483
--webserver_port=39337
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:31.535738 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:47037 (local address 127.8.13.252:33595)
0925 14:58:31.520470 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.520473 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.520481 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:31.521151 (+ 670us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.521152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.521165 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.521236 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.521883 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522540 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524139 (+ 1599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.524417 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524479 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.527278 (+ 2799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.527279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.527279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.527287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.534610 (+ 7323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.534975 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.534977 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.534977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.535046 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.535221 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.535222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.535222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.535336 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.535392 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.535547 (+ 155us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.535644 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260925 14:58:31.535959 15332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.520049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.8.13.193:47037)
0925 14:58:31.520222 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.520265 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.520827 (+ 562us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.521261 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.521262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.521531 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.521783 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.521789 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.522645 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.522645 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.523839 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.523842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.524476 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.524477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.524529 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.524946 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.524949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.526966 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.534699 (+ 7733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.534700 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.534701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.534911 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.535113 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.535113 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.535114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.535170 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.535391 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.535392 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.535434 (+ 42us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.535494 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.535640 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":92,"thread_start_us":83,"threads_started":1}
I20260925 14:58:31.536353 15327 heartbeater.cc:347] Connected to a master server at 127.8.13.252:33595
I20260925 14:58:31.536401 15327 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.536470 15327 heartbeater.cc:511] Master 127.8.13.252:33595 requested a full tablet report, sending...
I20260925 14:58:31.536824 15148 ts_manager.cc:194] Registered new tserver with Master: ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377)
W20260925 14:58:31.614705 15335 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:31.614949 15335 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:31.614976 15335 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:31.616251 15335 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:31.616307 15335 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:31.616344 15335 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:31.617816 15335 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:45079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=39337
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15335
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:31.618007 15335 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:31.618201 15335 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.622895 15340 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.622911 15341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.623029 15335 server_base.cc:1034] running on GCE node
W20260925 14:58:31.622912 15343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.623332 15335 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.623562 15335 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.624711 15335 hybrid_clock.cc:648] HybridClock initialized: now 1790348311624692 us; error 39 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.627014 15335 init.cc:377] Logged in from keytab as oryx/127.8.13.194@KRBTEST.COM (short username oryx)
I20260925 14:58:31.627461 15335 webserver.cc:460] Webserver started at http://127.8.13.194:39337/ using document root <none> and password file <none>
I20260925 14:58:31.627683 15335 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.627745 15335 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.629066 15335 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:31.629639 15350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.629783 15335 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:31.629864 15335 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "f854b76c9c734857bbf73a2741070116"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "971a00edaaf1d4b272442ce9322bcd00"
server_key_iv: "fdc25a45f5726b734155e075d26c80ab"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.630209 15335 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.650271 15335 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.652261 15335 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:31.652410 15335 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.652659 15335 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:31.652972 15335 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:31.653005 15335 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.653034 15335 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:31.653056 15335 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.658594 15335 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45079
I20260925 14:58:31.658692 15463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45079 every 8 connection(s)
I20260925 14:58:31.659000 15335 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:31.659749 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15335
I20260925 14:58:31.659904 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14693
I20260925 14:58:31.666368 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:39795
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:39483
--webserver_port=45689
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:31.671007 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.659915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:51745 (local address 127.8.13.252:33595)
0925 14:58:31.659985 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.659988 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.660770 (+ 782us) server_negotiation.cc:407] Connection header received
0925 14:58:31.661418 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.661419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.661432 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.661475 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.662702 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.663210 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.664947 (+ 1737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.665250 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.665293 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.667781 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.667783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.667783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.667791 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.669580 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.669989 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.669991 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.669991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.670045 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.670296 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.670297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.670297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.670427 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.670464 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.670826 (+ 362us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.670912 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
I20260925 14:58:31.671682 15468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.659920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.8.13.194:51745)
0925 14:58:31.660626 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.660676 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.661322 (+ 646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.661552 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.661556 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.662330 (+ 774us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.662614 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.662620 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.663294 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.663295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.664854 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.664858 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.665380 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.665381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.665520 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.666015 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.666025 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.667686 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.669643 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.669650 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.669652 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.669907 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.670106 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.670106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.670106 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.670233 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.670503 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.670509 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.670661 (+ 152us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.671233 (+ 572us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.671412 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":76,"threads_started":1}
I20260925 14:58:31.671840 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.660415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:51607 (local address 127.8.13.254:39483)
0925 14:58:31.660465 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.660467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.664067 (+ 3600us) server_negotiation.cc:407] Connection header received
0925 14:58:31.664101 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.664103 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.664115 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.664162 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.664484 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.665029 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.665985 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.666214 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.666274 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.668757 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.668759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.668759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.668767 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.670597 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.670983 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.670985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.670985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.671042 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.671228 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.671229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.671230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.671358 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.671397 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.671659 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.671745 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:58:31.672137 15464 heartbeater.cc:347] Connected to a master server at 127.8.13.252:33595
I20260925 14:58:31.672250 15464 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.672384 15469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.660250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:51607)
0925 14:58:31.663992 (+ 3742us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.664003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.664056 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.664164 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.664165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.664340 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.664406 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.664408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.665092 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.665093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.665667 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.665670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.666293 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.666293 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.666357 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.667289 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.667294 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.668673 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.670698 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.670699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.670699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.670896 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.671111 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.671111 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.671112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.671165 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.671441 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.671441 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.671514 (+ 73us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.671944 (+ 430us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.672280 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3722,"thread_start_us":66,"threads_started":1}
I20260925 14:58:31.672452 15464 heartbeater.cc:511] Master 127.8.13.252:33595 requested a full tablet report, sending...
I20260925 14:58:31.672729 15465 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:31.672775 15465 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.672832 15465 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:31.672905 15148 ts_manager.cc:194] Registered new tserver with Master: f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
I20260925 14:58:31.673183 14363 ts_manager.cc:194] Re-registered known tserver with Master: f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
I20260925 14:58:31.673658 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:51607
I20260925 14:58:31.675278 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.665157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:41475 (local address 127.8.13.253:33593)
0925 14:58:31.665213 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.665215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.667616 (+ 2401us) server_negotiation.cc:407] Connection header received
0925 14:58:31.667727 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.667728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.667741 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.667771 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.668108 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.668766 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.669469 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.669702 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.669762 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.672677 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.672679 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.672679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.672697 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.674184 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.674583 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.674584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.674585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.674644 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.674807 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.674808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.674809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.674916 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.674969 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.675104 (+ 135us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.675182 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 14:58:31.675288 15470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.664082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.8.13.194:41475)
0925 14:58:31.667521 (+ 3439us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.667561 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.667645 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.667831 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.667831 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.667977 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.668034 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.668035 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.668830 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.668830 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.669385 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.669387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.669819 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.669820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.669885 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.670219 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.670223 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.672141 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.674314 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.674315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.674316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.674512 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.674703 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.674704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.674704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.674758 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.674967 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.674968 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.675006 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.675076 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.675182 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3421,"thread_start_us":85,"threads_started":1}
I20260925 14:58:31.676959 15466 heartbeater.cc:347] Connected to a master server at 127.8.13.253:33593
I20260925 14:58:31.677008 15466 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.677080 15466 heartbeater.cc:511] Master 127.8.13.253:33593 requested a full tablet report, sending...
I20260925 14:58:31.677398 14961 ts_manager.cc:194] Registered new tserver with Master: f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
W20260925 14:58:31.750208 15471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:31.750433 15471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:31.750458 15471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:31.751662 15471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:31.751703 15471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:31.751729 15471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:31.753216 15471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:39795
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=45689
--tserver_master_addrs=127.8.13.254:39483,127.8.13.253:33593,127.8.13.252:33595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:31.753415 15471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:31.753614 15471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:31.758380 15476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:31.758440 15477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.758644 15471 server_base.cc:1034] running on GCE node
W20260925 14:58:31.758375 15479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:31.758872 15471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:31.759112 15471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:31.760258 15471 hybrid_clock.cc:648] HybridClock initialized: now 1790348311760245 us; error 29 us; skew 500 ppm
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:31.762530 15471 init.cc:377] Logged in from keytab as oryx/127.8.13.195@KRBTEST.COM (short username oryx)
I20260925 14:58:31.762955 15471 webserver.cc:460] Webserver started at http://127.8.13.195:45689/ using document root <none> and password file <none>
I20260925 14:58:31.763186 15471 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:31.763250 15471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:31.764559 15471 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:31.765077 15486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.765225 15471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:31.765303 15471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "59ca4a67ba77482c9c2c363f12d3055e"
format_stamp: "Formatted at 2026-09-25 14:58:30 on dist-test-slave-15hz"
server_key: "e63a5903120d9aa187e14de62583f572"
server_key_iv: "4025207698fea691d0b782bc269abcff"
server_key_version: "encryptionkey@0"
I20260925 14:58:31.765627 15471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:31.784991 15471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:31.787086 15471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:31.787230 15471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:31.787460 15471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:31.787791 15471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:31.787827 15471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.787856 15471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:31.787874 15471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:31.793315 15471 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:39795
I20260925 14:58:31.793370 15599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:39795 every 8 connection(s)
I20260925 14:58:31.793699 15471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.253@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.252@KRBTEST.COM
Sep 25 14:58:31 dist-test-slave-15hz krb5kdc[14318](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348311, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:31.802868 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15471
I20260925 14:58:31.805147 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:43883 (local address 127.8.13.252:33595)
0925 14:58:31.794830 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.794832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.794982 (+ 150us) server_negotiation.cc:407] Connection header received
0925 14:58:31.795905 (+ 923us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.795907 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.795918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.795960 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.797619 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.798161 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799003 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799291 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799353 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.802111 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.802112 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.802112 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.802120 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.803484 (+ 1364us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804004 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.804005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.804005 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.804077 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804352 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.804353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.804353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.804491 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.804533 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.804948 (+ 415us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.805056 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:31.805653 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:46905 (local address 127.8.13.253:33593)
0925 14:58:31.794495 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.794497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.794881 (+ 384us) server_negotiation.cc:407] Connection header received
0925 14:58:31.796011 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.796013 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.796023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.796063 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.796740 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.797537 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799066 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799354 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799407 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.801845 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.801846 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.801847 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.801855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.804431 (+ 2576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804851 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.804852 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.804852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.804905 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.805085 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.805086 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.805086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.805206 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.805260 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.805497 (+ 237us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.805570 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260925 14:58:31.805809 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:52997 (local address 127.8.13.254:39483)
0925 14:58:31.794483 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.794485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.794760 (+ 275us) server_negotiation.cc:407] Connection header received
0925 14:58:31.795841 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.795842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.795856 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.795899 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.796771 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.797512 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799055 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799254 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799287 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.802818 (+ 3531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.802820 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.802820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.802829 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.804346 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804746 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.804747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.804748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.804806 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804981 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.804982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.804982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.805105 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.805158 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.805644 (+ 486us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.805719 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:31.805806 15606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.8.13.195:46905)
0925 14:58:31.794805 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.794815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.795740 (+ 925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.796089 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.796090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.796421 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.796652 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.796657 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.797631 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.797632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.798940 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.798942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799455 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.799552 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.800235 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.800240 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.801728 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.804494 (+ 2766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804495 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.804495 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.804780 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.804967 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804968 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.804968 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.805015 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.805329 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.805330 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.805363 (+ 33us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.805419 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.805566 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":63,"thread_start_us":96,"threads_started":1}
I20260925 14:58:31.805914 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.8.13.195:43883)
0925 14:58:31.794914 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.794923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.795743 (+ 820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.795988 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.795990 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.796426 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.796654 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.796660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.798227 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.798228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.798932 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.798935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799372 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.799554 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.800171 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.800180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.802030 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.803608 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.803616 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.803618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.803909 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.804148 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.804149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.804279 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.804579 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.804585 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.804784 (+ 199us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.805465 (+ 681us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.805617 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":66,"thread_start_us":50,"threads_started":1}
I20260925 14:58:31.806134 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.794411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:52997)
0925 14:58:31.794630 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.794672 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.795735 (+ 1063us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.795985 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.795989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.796424 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:31.796654 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.796660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.797631 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.797632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.798940 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.798942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.799370 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.799372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.799553 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.800189 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:31.800193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:31.802700 (+ 2507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:31.804421 (+ 1721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804422 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.804423 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.804653 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.804869 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:31.804869 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:31.804870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:31.804919 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:31.805180 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:31.805181 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:31.805221 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.805291 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.805486 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":73,"spinlock_wait_cycles":7552,"thread_start_us":95,"threads_started":1}
I20260925 14:58:31.806470 15602 heartbeater.cc:347] Connected to a master server at 127.8.13.253:33593
I20260925 14:58:31.806677 15602 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.806943 15602 heartbeater.cc:511] Master 127.8.13.253:33593 requested a full tablet report, sending...
I20260925 14:58:31.807034 15600 heartbeater.cc:347] Connected to a master server at 127.8.13.252:33595
I20260925 14:58:31.807075 15600 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.807166 15600 heartbeater.cc:511] Master 127.8.13.252:33595 requested a full tablet report, sending...
I20260925 14:58:31.807225 15601 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39483
I20260925 14:58:31.807250 15601 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:31.807293 15601 heartbeater.cc:511] Master 127.8.13.254:39483 requested a full tablet report, sending...
I20260925 14:58:31.807427 14961 ts_manager.cc:194] Registered new tserver with Master: 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:31.807595 15148 ts_manager.cc:194] Registered new tserver with Master: 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:31.807838 14363 ts_manager.cc:194] Re-registered known tserver with Master: 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:31.808326 14363 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:52997
I20260925 14:58:31.813050 14421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.804677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50790 (local address 127.8.13.254:39483)
0925 14:58:31.805878 (+ 1201us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.805880 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.805888 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:31.805907 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.805908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.805917 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.805976 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.806896 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.807584 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.808641 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.808913 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.808954 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.810342 (+ 1388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.810343 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.810344 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.810350 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.811916 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.812298 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.812299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.812300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.812364 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.812547 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.812548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.812549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.812679 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.812780 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.812880 (+ 100us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.812949 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1187,"thread_start_us":95,"threads_started":1}
I20260925 14:58:31.814774 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.804103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36670 (local address 127.8.13.253:33593)
0925 14:58:31.807187 (+ 3084us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.807189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.807198 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:31.807217 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.807217 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.807230 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.808256 (+ 1026us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.808292 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.809020 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.809882 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.810179 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.810244 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.811381 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.811383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.811383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.811390 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.813448 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.813808 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.813809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.813810 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.813874 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.814257 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.814258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.814259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.814420 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.814506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.814592 (+ 86us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.814674 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3031,"thread_start_us":64,"threads_started":1}
I20260925 14:58:31.815233 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.805027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35912 (local address 127.8.13.252:33595)
0925 14:58:31.805222 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.805223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.808206 (+ 2983us) server_negotiation.cc:407] Connection header received
0925 14:58:31.808229 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.808230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.808240 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.808282 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:31.809163 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.809760 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.811065 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.811298 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.811336 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.812656 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:31.812657 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:31.812657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:31.812664 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:31.814120 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.814425 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.814426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.814426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.814495 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:31.814686 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:31.814687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:31.814688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:31.814818 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:31.814886 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.815097 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.815163 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":49,"threads_started":1}
I20260925 14:58:31.814679 14363 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:31.816705 14363 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:31.824440 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.8.13.193:40377)
0925 14:58:31.820682 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.820685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.820695 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:31.820733 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.820736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.820752 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.820965 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:31.821013 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.822118 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.823386 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824098 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824165 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824243 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.824316 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.824354 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":82,"threads_started":1}
I20260925 14:58:31.824429 14921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:40377 (local address 127.0.0.1:44862)
0925 14:58:31.820275 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.820287 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.820331 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.820852 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.820853 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.820855 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:31.820903 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.820904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.822193 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.822194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.823317 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.823319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824191 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.824251 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824256 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.824303 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.824348 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":10}
I20260925 14:58:31.824569 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45079 (local address 127.0.0.1:49276)
0925 14:58:31.820770 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.820780 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.820828 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.821081 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.821081 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.821084 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:31.821129 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.821131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.822064 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.822065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.823323 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.823324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824372 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.824431 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824434 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.824477 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.824511 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":34,"thread_start_us":106,"threads_started":1}
I20260925 14:58:31.824594 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39666 (local address 127.8.13.195:39795)
0925 14:58:31.820972 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.820977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.821105 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:58:31.821151 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.821153 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.821169 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.821246 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:31.821327 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.822367 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.823335 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824097 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824165 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824238 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.824474 (+ 236us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.824514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":95,"threads_started":1}
I20260925 14:58:31.824776 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39795 (local address 127.0.0.1:39666)
0925 14:58:31.821026 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.821035 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.821094 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.821240 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.821240 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.821242 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:31.821271 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.821272 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.822436 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.822437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.823256 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.823258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824272 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.824334 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824339 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.824383 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.824421 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"thread_start_us":66,"threads_started":1}
I20260925 14:58:31.824815 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.820487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49276 (local address 127.8.13.194:45079)
0925 14:58:31.820639 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.820644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.820880 (+ 236us) server_negotiation.cc:407] Connection header received
0925 14:58:31.820924 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.820926 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.820942 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.821012 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:31.821183 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.821996 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.823609 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.824214 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.824271 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.824342 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.824680 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.824742 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
I20260925 14:58:31.825423 15534 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 (DEFAULT_TABLE table=test-table [id=d57a9e7bf22243b1ae315fc34bda8991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:31.825505 15398 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 (DEFAULT_TABLE table=test-table [id=d57a9e7bf22243b1ae315fc34bda8991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:31.825801 15398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcc7bda16c4a569303e23e982c2ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:31.825807 15534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcc7bda16c4a569303e23e982c2ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:31.825713 15261 tablet_service.cc:1467] Processing CreateTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 (DEFAULT_TABLE table=test-table [id=d57a9e7bf22243b1ae315fc34bda8991]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:31.825978 15261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddbcc7bda16c4a569303e23e982c2ed1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:31.828547 15626 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Bootstrap starting.
I20260925 14:58:31.829181 15627 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Bootstrap starting.
I20260925 14:58:31.829535 15626 tablet_bootstrap.cc:654] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:31.829782 15627 tablet_bootstrap.cc:654] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:31.829974 15626 log.cc:824] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:31.830085 15627 log.cc:824] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:31.830828 15627 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: No bootstrap required, opened a new log
I20260925 14:58:31.830896 15627 ts_tablet_manager.cc:1397] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:31.831303 15626 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: No bootstrap required, opened a new log
I20260925 14:58:31.831465 15626 ts_tablet_manager.cc:1397] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:31.833256 15630 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Bootstrap starting.
I20260925 14:58:31.833532 15627 raft_consensus.cc:348] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.833699 15627 raft_consensus.cc:374] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:31.833798 15627 raft_consensus.cc:729] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9f8f145a504cea8ac76a5d36f2dab3, State: Initialized, Role: FOLLOWER
I20260925 14:58:31.833811 15630 tablet_bootstrap.cc:654] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:31.833920 15627 consensus_queue.cc:260] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.834156 15630 log.cc:824] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:31.834172 15627 ts_tablet_manager.cc:1428] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:31.834635 15328 heartbeater.cc:503] Master 127.8.13.254:39483 was elected leader, sending a full tablet report...
I20260925 14:58:31.835851 15630 tablet_bootstrap.cc:492] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: No bootstrap required, opened a new log
I20260925 14:58:31.835925 15630 ts_tablet_manager.cc:1397] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:31.836215 15626 raft_consensus.cc:348] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.836376 15626 raft_consensus.cc:374] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:31.836406 15626 raft_consensus.cc:729] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f854b76c9c734857bbf73a2741070116, State: Initialized, Role: FOLLOWER
I20260925 14:58:31.836524 15626 consensus_queue.cc:260] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.836769 15626 ts_tablet_manager.cc:1428] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260925 14:58:31.836786 15465 heartbeater.cc:503] Master 127.8.13.254:39483 was elected leader, sending a full tablet report...
I20260925 14:58:31.838963 15630 raft_consensus.cc:348] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.839095 15630 raft_consensus.cc:374] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:31.839128 15630 raft_consensus.cc:729] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ca4a67ba77482c9c2c363f12d3055e, State: Initialized, Role: FOLLOWER
I20260925 14:58:31.839244 15630 consensus_queue.cc:260] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:31.839453 15630 ts_tablet_manager.cc:1428] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:31.839607 15601 heartbeater.cc:503] Master 127.8.13.254:39483 was elected leader, sending a full tablet report...
I20260925 14:58:31.847519 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.845336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35922 (local address 127.8.13.252:33595)
0925 14:58:31.845389 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:31.845391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:31.845398 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:31.845437 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:31.845438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:31.845457 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:31.845506 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:31.845596 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.846069 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.846888 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.847290 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.847345 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.847399 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:31.847438 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 14:58:31.847463 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:31.847546 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:31.845272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35922)
0925 14:58:31.845332 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:31.845344 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:31.845388 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:31.845510 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:31.845510 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:31.845512 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:31.845556 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.845558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.846116 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.846116 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.846811 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:31.846812 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:31.847351 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:31.847352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:31.847395 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:31.847398 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:31.847434 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 14:58:31.847467 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 14:58:31.847942 15158 raft_consensus.cc:1203] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260925 14:58:31.848763 15187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3577c8dc9c9a4d8bbd77055ee1786b16. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:31.848865 15187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: LEARNER
W20260925 14:58:31.909881 15467 tablet.cc:2369] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:32.020196 15330 tablet.cc:2369] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:32.045055 15603 tablet.cc:2369] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:32.126022 15634 raft_consensus.cc:482] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:32.126170 15634 raft_consensus.cc:504] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:32.126521 15634 leader_election.cc:288] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377), f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
I20260925 14:58:32.130141 15621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.126855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:36281 (local address 127.8.13.193:40377)
0925 14:58:32.126921 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.126923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.126934 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:32.126978 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.126978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.126990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.127059 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:32.127244 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.127828 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129198 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.129778 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129846 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.129855 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.130033 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.130070 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 14:58:32.130151 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.126923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:36233 (local address 127.8.13.194:45079)
0925 14:58:32.126962 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.126964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.126972 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:32.126996 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.126997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.127006 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.127051 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:32.127212 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.127927 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129198 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.129778 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129846 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.129855 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.130033 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.130070 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:32.130162 15604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.126819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45079 (local address 127.8.13.195:36233)
0925 14:58:32.126892 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:32.126900 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:32.126925 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:32.127062 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:32.127062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:32.127064 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:32.127135 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:32.127138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.128049 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.128050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:32.129093 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:32.129096 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.129867 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:32.129961 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.129967 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:58:32.130029 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:58:32.130069 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":7}
I20260925 14:58:32.130168 15605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.126702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:40377 (local address 127.8.13.195:36281)
0925 14:58:32.126754 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:32.126766 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:32.126816 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:32.127111 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:32.127137 (+ 26us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:32.127138 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:32.127188 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:32.127189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.127909 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.127910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:32.129093 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:32.129095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.129867 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.129868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:32.129961 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.129967 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:58:32.130020 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 14:58:32.130069 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":6}
I20260925 14:58:32.130586 15281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcc7bda16c4a569303e23e982c2ed1" candidate_uuid: "59ca4a67ba77482c9c2c363f12d3055e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" is_pre_election: true
I20260925 14:58:32.130596 15418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcc7bda16c4a569303e23e982c2ed1" candidate_uuid: "59ca4a67ba77482c9c2c363f12d3055e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f854b76c9c734857bbf73a2741070116" is_pre_election: true
I20260925 14:58:32.130753 15418 raft_consensus.cc:2454] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ca4a67ba77482c9c2c363f12d3055e in term 0.
I20260925 14:58:32.130753 15281 raft_consensus.cc:2454] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59ca4a67ba77482c9c2c363f12d3055e in term 0.
I20260925 14:58:32.130977 15490 leader_election.cc:302] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ca4a67ba77482c9c2c363f12d3055e, f854b76c9c734857bbf73a2741070116; no voters:
I20260925 14:58:32.131093 15634 raft_consensus.cc:2781] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:32.131146 15634 raft_consensus.cc:482] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:32.131178 15634 raft_consensus.cc:3037] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:32.131978 15634 raft_consensus.cc:504] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:32.132093 15634 leader_election.cc:288] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [CANDIDATE]: Term 1 election: Requested vote from peers ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377), f854b76c9c734857bbf73a2741070116 (127.8.13.194:45079)
I20260925 14:58:32.132248 15281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcc7bda16c4a569303e23e982c2ed1" candidate_uuid: "59ca4a67ba77482c9c2c363f12d3055e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
I20260925 14:58:32.132287 15418 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddbcc7bda16c4a569303e23e982c2ed1" candidate_uuid: "59ca4a67ba77482c9c2c363f12d3055e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f854b76c9c734857bbf73a2741070116"
I20260925 14:58:32.132328 15281 raft_consensus.cc:3037] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:32.132347 15418 raft_consensus.cc:3037] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:32.133189 15418 raft_consensus.cc:2454] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ca4a67ba77482c9c2c363f12d3055e in term 1.
I20260925 14:58:32.133188 15281 raft_consensus.cc:2454] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59ca4a67ba77482c9c2c363f12d3055e in term 1.
I20260925 14:58:32.133378 15490 leader_election.cc:302] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ca4a67ba77482c9c2c363f12d3055e, f854b76c9c734857bbf73a2741070116; no voters:
I20260925 14:58:32.133463 15634 raft_consensus.cc:2781] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:32.133595 15634 raft_consensus.cc:686] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 LEADER]: Becoming Leader. State: Replica: 59ca4a67ba77482c9c2c363f12d3055e, State: Running, Role: LEADER
I20260925 14:58:32.133682 15634 consensus_queue.cc:237] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:32.134538 14362 catalog_manager.cc:5495] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e reported cstate change: term changed from 0 to 1, leader changed from <none> to 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195). New cstate: current_term: 1 leader_uuid: "59ca4a67ba77482c9c2c363f12d3055e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:32.135138 14923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: NON_VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: true } } }
I20260925 14:58:32.135238 14923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:32.136018 14923 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: attempting to promote NON_VOTER e4576b5c4e5d4f8c98c4327ca7fc1dd0 to VOTER
I20260925 14:58:32.136128 14923 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:32.136526 14971 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Refusing update from remote peer 3577c8dc9c9a4d8bbd77055ee1786b16: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 14:58:32.136534 15158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 LEARNER]: Refusing update from remote peer 3577c8dc9c9a4d8bbd77055ee1786b16: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260925 14:58:32.136718 15642 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260925 14:58:32.136832 15641 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260925 14:58:32.137521 15642 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.137708 14971 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.138558 15642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.138634 15642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:32.138624 15114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.138721 15114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:32.138768 15114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.138813 15114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:32.138865 15158 raft_consensus.cc:2932] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.139622 15187 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.139636 15644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:32.139714 15187 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:32.139721 15644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:32.176927 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.174704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.8.13.195:39795)
0925 14:58:32.174766 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.174768 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.174776 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:32.174809 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.174810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.174829 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.174885 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:32.175012 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.175476 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.176219 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.176406 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.176459 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.176605 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:32.176715 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:32.176787 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.176823 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.176845 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:32.178644 15281 raft_consensus.cc:1261] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Refusing update from remote peer 59ca4a67ba77482c9c2c363f12d3055e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:32.178653 15418 raft_consensus.cc:1261] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Refusing update from remote peer 59ca4a67ba77482c9c2c363f12d3055e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:32.179005 15639 consensus_queue.cc:1035] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [LEADER]: Connected to new peer: Peer: permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:32.179149 15634 consensus_queue.cc:1035] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:32.182010 15649 mvcc.cc:205] Tried to move back new op lower bound from 7333266686681350144 to 7333266686500184064. Current Snapshot: MvccSnapshot[applied={T|T < 7333266686681350144}]
I20260925 14:58:32.183913 15648 mvcc.cc:205] Tried to move back new op lower bound from 7333266686681350144 to 7333266686500184064. Current Snapshot: MvccSnapshot[applied={T|T < 7333266686681350144 or (T in {7333266686681350144})}]
I20260925 14:58:32.184252 14362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:32.184326 14362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:32.185590 14362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3577c8dc9c9a4d8bbd77055ee1786b16: Sending DeleteTablet for 3 replicas of tablet ddbcc7bda16c4a569303e23e982c2ed1
I20260925 14:58:32.185978 15398 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:32 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:49276
I20260925 14:58:32.186013 15534 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:32 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:39666
I20260925 14:58:32.186074 15261 tablet_service.cc:1514] Processing DeleteTablet for tablet ddbcc7bda16c4a569303e23e982c2ed1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:32 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:44862
I20260925 14:58:32.186200 15658 tablet_replica.cc:330] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: stopping tablet replica
I20260925 14:58:32.186286 15658 raft_consensus.cc:2229] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:32.186378 15658 raft_consensus.cc:2258] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:32.186707 15659 tablet_replica.cc:330] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: stopping tablet replica
I20260925 14:58:32.187034 15658 ts_tablet_manager.cc:1905] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:32.187204 15659 raft_consensus.cc:2229] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:32.187259 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14331
I20260925 14:58:32.188663 15659 raft_consensus.cc:2258] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:32.190971 15659 ts_tablet_manager.cc:1905] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:32.192790 15658 ts_tablet_manager.cc:1918] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:32.192922 15658 log.cc:1196] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/ddbcc7bda16c4a569303e23e982c2ed1
I20260925 14:58:32.193336 15658 ts_tablet_manager.cc:1939] T ddbcc7bda16c4a569303e23e982c2ed1 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting consensus metadata
I20260925 14:58:32.193490 15659 ts_tablet_manager.cc:1918] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:32.193615 15659 log.cc:1196] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/ddbcc7bda16c4a569303e23e982c2ed1
W20260925 14:58:32.194021 15489 connection.cc:441] server connection from 127.0.0.1:39666 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39666 (request call id 1) could send its response
I20260925 14:58:32.194237 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.186630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35926 (local address 127.8.13.252:33595)
0925 14:58:32.186670 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.186672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.186757 (+ 85us) server_negotiation.cc:407] Connection header received
0925 14:58:32.186845 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.186846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.186856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.186883 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:32.187671 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.188406 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.189411 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.189688 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.189726 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.191330 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:32.191331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:32.191332 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:32.191340 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:32.192969 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.193381 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.193383 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.193383 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.193441 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.193642 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.193643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.193644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.193774 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:32.193848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.194054 (+ 206us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.194151 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:32.194835 15659 ts_tablet_manager.cc:1939] T ddbcc7bda16c4a569303e23e982c2ed1 P ed9f8f145a504cea8ac76a5d36f2dab3: Deleting consensus metadata
W20260925 14:58:32.195293 15216 connection.cc:441] server connection from 127.0.0.1:44862 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44862 (request call id 1) could send its response
I20260925 14:58:32.195322 15657 tablet_replica.cc:330] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: stopping tablet replica
I20260925 14:58:32.195523 15657 raft_consensus.cc:2229] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:32.195724 15657 raft_consensus.cc:2258] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:32.196264 15657 ts_tablet_manager.cc:1905] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:32.197924 15657 ts_tablet_manager.cc:1918] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:32.198060 15657 log.cc:1196] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/ddbcc7bda16c4a569303e23e982c2ed1
I20260925 14:58:32.198357 15657 ts_tablet_manager.cc:1939] T ddbcc7bda16c4a569303e23e982c2ed1 P f854b76c9c734857bbf73a2741070116: Deleting consensus metadata
W20260925 14:58:32.198778 15351 connection.cc:441] server connection from 127.0.0.1:49276 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:49276 (request call id 1) could send its response
I20260925 14:58:32.203311 15333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.195422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35934 (local address 127.8.13.252:33595)
0925 14:58:32.195455 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.195457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.195822 (+ 365us) server_negotiation.cc:407] Connection header received
0925 14:58:32.195914 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.195915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.195925 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.195954 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:32.196929 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.197616 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.198573 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.198887 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.198940 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.200746 (+ 1806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:32.200747 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:32.200747 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:32.200755 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:32.202075 (+ 1320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.202428 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.202430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.202430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.202487 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.202696 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.202697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.202697 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.202838 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:32.202902 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.203142 (+ 240us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.203217 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 14:58:32.203686 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:32.195747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36682 (local address 127.8.13.253:33593)
0925 14:58:32.195791 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:32.195793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:32.195915 (+ 122us) server_negotiation.cc:407] Connection header received
0925 14:58:32.196002 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:32.196003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:32.196016 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:32.196043 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:32.196673 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.197478 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.198574 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:32.198898 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:32.198940 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:32.200745 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:32.200746 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:32.200747 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:32.200756 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:32.202358 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.202816 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.202817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.202818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.202882 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:32.203059 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:32.203060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:32.203060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:32.203194 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:32.203280 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:32.203474 (+ 194us) server_negotiation.cc:299] Negotiation successful
0925 14:58:32.203590 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:58:33.182839 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.182574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:60439)
0925 14:58:33.182699 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.182783 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
W20260925 14:58:33.183092 15328 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:39483 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.183837 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.183585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:46697)
0925 14:58:33.183717 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.183791 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260925 14:58:33.184092 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.183828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:42497)
0925 14:58:33.183982 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.184049 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":59,"threads_started":1}
W20260925 14:58:33.184175 15601 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:39483 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
W20260925 14:58:33.184358 15465 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:39483 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.685868 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3577c8dc9c9a4d8bbd77055ee1786b16)
I20260925 14:58:33.686029 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:33.686105 15671 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 3577c8dc9c9a4d8bbd77055ee1786b16)
I20260925 14:58:33.686167 15671 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:33.686329 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595)
I20260925 14:58:33.686342 15671 leader_election.cc:288] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253:33593)
I20260925 14:58:33.686694 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51256)
0925 14:58:33.686571 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.686651 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":52,"threads_started":1}
I20260925 14:58:33.686715 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51254)
0925 14:58:33.686631 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.686678 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":42,"threads_started":1}
W20260925 14:58:33.686875 15134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
W20260925 14:58:33.686926 14947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.687419 15674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.687185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51274)
0925 14:58:33.687339 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.687375 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260925 14:58:33.687484 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.687385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51288)
0925 14:58:33.687422 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.687457 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 14:58:33.687493 15134 leader_election.cc:334] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
W20260925 14:58:33.687644 14947 leader_election.cc:334] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.689491 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:33593 (local address 127.0.0.1:36698)
0925 14:58:33.686748 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.686758 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:33.686817 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:33.687078 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:33.687079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:33.687084 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:33.687148 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:33.687150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.688010 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.688011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:33.688783 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:33.688785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.689302 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.689303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:33.689356 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.689361 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:33.689406 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:58:33.689444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":42,"threads_started":1}
I20260925 14:58:33.689530 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.8.13.253:33593)
0925 14:58:33.686925 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:33.686927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:33.686934 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:33.686956 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:33.686956 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:33.686966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:33.687209 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:33.687242 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.687934 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.688864 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.689236 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.689288 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.689297 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:33.689425 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 14:58:33.689468 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":47,"threads_started":1}
I20260925 14:58:33.689607 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35940 (local address 127.8.13.252:33595)
0925 14:58:33.687016 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:33.687018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:33.687025 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:33.687046 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:33.687047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:33.687056 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:33.687278 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:33.687312 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.688078 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.688944 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.689326 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.689387 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.689396 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:33.689522 (+ 126us) server_negotiation.cc:299] Negotiation successful
0925 14:58:33.689551 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":47,"threads_started":1}
I20260925 14:58:33.689771 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.686593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:33595 (local address 127.0.0.1:35940)
0925 14:58:33.686763 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.686772 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:33.686827 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:33.687157 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:33.687158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:33.687161 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:33.687218 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:33.687220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.688151 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.688152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:33.688872 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:33.688873 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:33.689393 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:33.689394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:33.689451 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:33.689455 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:33.689505 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 14:58:33.689537 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":46,"threads_started":1}
I20260925 14:58:33.689843 14971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "153aff0e44394f4a9f68cfb489c275ab" is_pre_election: true
I20260925 14:58:33.689940 14971 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4576b5c4e5d4f8c98c4327ca7fc1dd0 in term 1.
I20260925 14:58:33.689996 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153aff0e44394f4a9f68cfb489c275ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" is_pre_election: true
I20260925 14:58:33.690089 15135 leader_election.cc:302] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153aff0e44394f4a9f68cfb489c275ab, e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters: 3577c8dc9c9a4d8bbd77055ee1786b16
I20260925 14:58:33.690117 15158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153aff0e44394f4a9f68cfb489c275ab in term 1.
I20260925 14:58:33.690179 15671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:58:33.690212 15671 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3577c8dc9c9a4d8bbd77055ee1786b16)
I20260925 14:58:33.690228 15671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:58:33.690248 14947 leader_election.cc:302] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153aff0e44394f4a9f68cfb489c275ab, e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters: 3577c8dc9c9a4d8bbd77055ee1786b16
I20260925 14:58:33.690352 15670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260925 14:58:33.690399 15670 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Starting leader election (detected failure of leader 3577c8dc9c9a4d8bbd77055ee1786b16)
I20260925 14:58:33.690423 15670 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:58:33.691097 15671 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:33.691145 15670 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:33.691238 15670 leader_election.cc:288] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 election: Requested vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595)
I20260925 14:58:33.691247 15671 leader_election.cc:288] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 election: Requested vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), 153aff0e44394f4a9f68cfb489c275ab (127.8.13.253:33593)
I20260925 14:58:33.691447 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.691354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51296)
0925 14:58:33.691391 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.691423 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 14:58:33.691455 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.691354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51294)
0925 14:58:33.691415 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.691437 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 14:58:33.691453 14971 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "153aff0e44394f4a9f68cfb489c275ab"
I20260925 14:58:33.691475 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153aff0e44394f4a9f68cfb489c275ab" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
I20260925 14:58:33.691588 14971 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate e4576b5c4e5d4f8c98c4327ca7fc1dd0 in current term 2: Already voted for candidate 153aff0e44394f4a9f68cfb489c275ab in this term.
I20260925 14:58:33.691588 15158 raft_consensus.cc:2379] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 153aff0e44394f4a9f68cfb489c275ab in current term 2: Already voted for candidate e4576b5c4e5d4f8c98c4327ca7fc1dd0 in this term.
I20260925 14:58:33.691908 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.691815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51320)
0925 14:58:33.691856 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.691883 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:58:33.691977 15134 leader_election.cc:334] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.692009 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:33.691901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51332)
0925 14:58:33.691964 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:33.691989 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260925 14:58:33.692019 15134 leader_election.cc:302] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters: 153aff0e44394f4a9f68cfb489c275ab, 3577c8dc9c9a4d8bbd77055ee1786b16
I20260925 14:58:33.692090 15671 raft_consensus.cc:2726] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260925 14:58:33.692093 14947 leader_election.cc:334] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:33.692162 14947 leader_election.cc:302] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 153aff0e44394f4a9f68cfb489c275ab; no voters: 3577c8dc9c9a4d8bbd77055ee1786b16, e4576b5c4e5d4f8c98c4327ca7fc1dd0
I20260925 14:58:33.692221 15670 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260925 14:58:34.183785 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:34.183513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:59077)
0925 14:58:34.183692 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:34.183737 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":117,"threads_started":1}
I20260925 14:58:34.184804 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:34.184583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:53759)
0925 14:58:34.184717 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:34.184757 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
I20260925 14:58:34.184845 15682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:34.184681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:58199)
0925 14:58:34.184772 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:34.184802 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77,"thread_start_us":56,"threads_started":1}
I20260925 14:58:35.184579 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.184306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:34759)
0925 14:58:35.184481 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.184522 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":68,"threads_started":1}
I20260925 14:58:35.185412 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.185230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:57155)
0925 14:58:35.185327 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.185367 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":51,"threads_started":1}
I20260925 14:58:35.185418 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.185230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:41973)
0925 14:58:35.185345 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.185376 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":54,"threads_started":1}
I20260925 14:58:35.251752 15686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:35.251856 15686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:35.252023 15686 leader_election.cc:288] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595)
I20260925 14:58:35.252395 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.252180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51568)
0925 14:58:35.252290 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.252339 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":61,"threads_started":1}
I20260925 14:58:35.252440 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153aff0e44394f4a9f68cfb489c275ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" is_pre_election: true
I20260925 14:58:35.252554 15158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 153aff0e44394f4a9f68cfb489c275ab in term 2.
I20260925 14:58:35.252755 14947 leader_election.cc:302] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 153aff0e44394f4a9f68cfb489c275ab, e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters:
I20260925 14:58:35.252791 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.252715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51572)
0925 14:58:35.252743 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.252770 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260925 14:58:35.252844 15686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260925 14:58:35.252884 15686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:35.252902 15686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 2 FOLLOWER]: Advancing to term 3
W20260925 14:58:35.252909 14947 leader_election.cc:334] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:35.253595 15686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:35.253701 15686 leader_election.cc:288] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 election: Requested vote from peers 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483), e4576b5c4e5d4f8c98c4327ca7fc1dd0 (127.8.13.252:33595)
I20260925 14:58:35.254026 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.253826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51576)
0925 14:58:35.253907 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.253942 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260925 14:58:35.254053 15158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "153aff0e44394f4a9f68cfb489c275ab" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0"
I20260925 14:58:35.254153 15158 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 2 FOLLOWER]: Advancing to term 3
I20260925 14:58:35.254397 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.254315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51592)
0925 14:58:35.254354 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.254379 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 14:58:35.254464 14947 leader_election.cc:334] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
I20260925 14:58:35.254905 15158 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 153aff0e44394f4a9f68cfb489c275ab in term 3.
I20260925 14:58:35.255098 14947 leader_election.cc:302] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 153aff0e44394f4a9f68cfb489c275ab, e4576b5c4e5d4f8c98c4327ca7fc1dd0; no voters: 3577c8dc9c9a4d8bbd77055ee1786b16
I20260925 14:58:35.255172 15686 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 3 FOLLOWER]: Leader election won for term 3
I20260925 14:58:35.255271 15686 raft_consensus.cc:686] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [term 3 LEADER]: Becoming Leader. State: Replica: 153aff0e44394f4a9f68cfb489c275ab, State: Running, Role: LEADER
I20260925 14:58:35.255329 15686 consensus_queue.cc:237] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } }
I20260925 14:58:35.255679 15689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153aff0e44394f4a9f68cfb489c275ab. Latest consensus state: current_term: 3 leader_uuid: "153aff0e44394f4a9f68cfb489c275ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:35.255784 15689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:35.255951 15691 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:35.256326 15691 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:35.256388 15691 catalog_manager.cc:1259] Loaded cluster ID: 9d73cb0257cd4c7d84a0432090fe0440
I20260925 14:58:35.256418 15691 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:35.256600 15691 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:35.256672 15691 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Loaded TSK: 0
I20260925 14:58:35.256804 15691 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:35.283015 14960 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36682:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:35.284740 14960 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:35.285741 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.285649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51610)
0925 14:58:35.285683 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.285712 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 14:58:35.285748 15158 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [term 3 FOLLOWER]: Refusing update from remote peer 153aff0e44394f4a9f68cfb489c275ab: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260925 14:58:35.286062 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.285985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.0.0.1:51626)
0925 14:58:35.286023 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.286045 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 14:58:35.286074 15686 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260925 14:58:35.286212 14947 consensus_peers.cc:487] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab -> Peer 3577c8dc9c9a4d8bbd77055ee1786b16 (127.8.13.254:39483): Couldn't send request to peer 3577c8dc9c9a4d8bbd77055ee1786b16. Status: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260925 14:58:35.286692 15693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 153aff0e44394f4a9f68cfb489c275ab. Latest consensus state: current_term: 3 leader_uuid: "153aff0e44394f4a9f68cfb489c275ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:35.286783 15693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:35.287223 15689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "153aff0e44394f4a9f68cfb489c275ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:35.287312 15689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:35.287472 15693 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "153aff0e44394f4a9f68cfb489c275ab" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3577c8dc9c9a4d8bbd77055ee1786b16" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39483 } } peers { permanent_uuid: "153aff0e44394f4a9f68cfb489c275ab" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 33593 } attrs { promote: false } } peers { permanent_uuid: "e4576b5c4e5d4f8c98c4327ca7fc1dd0" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 33595 } attrs { promote: false } } }
I20260925 14:58:35.287580 15693 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4576b5c4e5d4f8c98c4327ca7fc1dd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 14:58:35.292236 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.288905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49290 (local address 127.8.13.194:45079)
0925 14:58:35.289376 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.289378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.289386 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:35.289412 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.289413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.289423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.289483 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:35.289655 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290470 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.291308 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.291909 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.291973 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.291997 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.292118 (+ 121us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.292162 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":64,"threads_started":1}
I20260925 14:58:35.292232 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.288832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45079 (local address 127.0.0.1:49290)
0925 14:58:35.289038 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.289048 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:35.289110 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:35.289490 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:35.289491 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:35.289492 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:35.289540 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.289542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290524 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.290525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.291229 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.291231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.291991 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.291992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.292059 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.292063 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:35.292112 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:35.292155 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184}
I20260925 14:58:35.292304 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.288918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:40377 (local address 127.0.0.1:44866)
0925 14:58:35.289162 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.289172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:35.289228 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:35.289357 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:35.289358 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:35.289359 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:35.289409 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.289411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290229 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.290230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.291309 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.291311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.292045 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.292045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.292109 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.292113 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:35.292160 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:35.292202 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":82,"threads_started":1}
I20260925 14:58:35.292470 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.289001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.8.13.193:40377)
0925 14:58:35.289108 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.289110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.289223 (+ 113us) server_negotiation.cc:407] Connection header received
0925 14:58:35.289286 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.289287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.289296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.289350 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:35.289480 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290157 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.291386 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.291954 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.292039 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.292052 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.292263 (+ 211us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.292405 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":53,"threads_started":1}
I20260925 14:58:35.292490 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.289076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39795 (local address 127.0.0.1:39690)
0925 14:58:35.289181 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.289191 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:35.289228 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:35.289579 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:35.289580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:35.289581 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:35.289609 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.289610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290459 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.290460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.291487 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.291489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.292287 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.292288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.292347 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.292350 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:35.292395 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:58:35.292432 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
I20260925 14:58:35.292650 15261 tablet_service.cc:1467] Processing CreateTablet for tablet 0a073adff209463facc7abcc15743c59 (DEFAULT_TABLE table=test-table [id=d38f136c5fa54db48517d1dea44b8734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:35.292722 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.289357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.8.13.195:39795)
0925 14:58:35.289468 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.289470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.289478 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:35.289501 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.289501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.289512 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.289589 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:35.289683 (+ 94us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.290384 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.291589 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.292187 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.292243 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.292256 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.292604 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.292652 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":62,"threads_started":1}
I20260925 14:58:35.292771 15261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a073adff209463facc7abcc15743c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:35.292903 15534 tablet_service.cc:1467] Processing CreateTablet for tablet 0a073adff209463facc7abcc15743c59 (DEFAULT_TABLE table=test-table [id=d38f136c5fa54db48517d1dea44b8734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:35.293020 15534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a073adff209463facc7abcc15743c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:35.293015 15398 tablet_service.cc:1467] Processing CreateTablet for tablet 0a073adff209463facc7abcc15743c59 (DEFAULT_TABLE table=test-table [id=d38f136c5fa54db48517d1dea44b8734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:35.293109 15398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a073adff209463facc7abcc15743c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:35.294421 15705 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: Bootstrap starting.
I20260925 14:58:35.294524 15703 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Bootstrap starting.
I20260925 14:58:35.294646 15704 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: Bootstrap starting.
I20260925 14:58:35.294813 15705 tablet_bootstrap.cc:654] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:35.294934 15704 tablet_bootstrap.cc:654] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:35.295384 15705 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: No bootstrap required, opened a new log
I20260925 14:58:35.295430 15705 ts_tablet_manager.cc:1397] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:35.295499 15703 tablet_bootstrap.cc:654] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:35.295500 15704 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: No bootstrap required, opened a new log
I20260925 14:58:35.295578 15704 ts_tablet_manager.cc:1397] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:35.295645 15705 raft_consensus.cc:348] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.295720 15705 raft_consensus.cc:374] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:35.295739 15705 raft_consensus.cc:729] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f854b76c9c734857bbf73a2741070116, State: Initialized, Role: FOLLOWER
I20260925 14:58:35.295776 15705 consensus_queue.cc:260] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.295745 15704 raft_consensus.cc:348] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.295820 15704 raft_consensus.cc:374] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:35.295864 15704 raft_consensus.cc:729] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed9f8f145a504cea8ac76a5d36f2dab3, State: Initialized, Role: FOLLOWER
I20260925 14:58:35.295935 15704 consensus_queue.cc:260] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.295967 15705 ts_tablet_manager.cc:1428] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:35.296128 15703 tablet_bootstrap.cc:492] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: No bootstrap required, opened a new log
I20260925 14:58:35.296181 15703 ts_tablet_manager.cc:1397] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 14:58:35.296288 15703 raft_consensus.cc:348] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.296362 15703 raft_consensus.cc:374] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:35.296386 15703 raft_consensus.cc:729] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59ca4a67ba77482c9c2c363f12d3055e, State: Initialized, Role: FOLLOWER
I20260925 14:58:35.296422 15703 consensus_queue.cc:260] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.296525 15704 ts_tablet_manager.cc:1428] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:35.296582 15703 ts_tablet_manager.cc:1428] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:35.296648 15466 heartbeater.cc:503] Master 127.8.13.253:33593 was elected leader, sending a full tablet report...
I20260925 14:58:35.296936 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.296795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:34923)
0925 14:58:35.296878 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.296908 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260925 14:58:35.296944 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.296852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:42009)
0925 14:58:35.296892 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.296917 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 14:58:35.297253 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.297120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:45937)
0925 14:58:35.297199 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.297225 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260925 14:58:35.297274 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.297197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:52103)
0925 14:58:35.297231 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.297251 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:58:35.297544 15602 heartbeater.cc:503] Master 127.8.13.253:33593 was elected leader, sending a full tablet report...
I20260925 14:58:35.297638 15329 heartbeater.cc:503] Master 127.8.13.253:33593 was elected leader, sending a full tablet report...
I20260925 14:58:35.297917 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.297840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:49949)
0925 14:58:35.297868 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.297891 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260925 14:58:35.298056 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.297953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:34615)
0925 14:58:35.298012 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.298033 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260925 14:58:35.573410 15709 raft_consensus.cc:482] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:35.573509 15709 raft_consensus.cc:504] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.573815 15709 leader_election.cc:288] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377), 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:35.577694 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.574317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:34157 (local address 127.8.13.195:39795)
0925 14:58:35.574366 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.574367 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.574405 (+ 38us) server_negotiation.cc:407] Connection header received
0925 14:58:35.574474 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.574475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.574487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.574538 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:35.574657 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.575328 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.576763 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.577321 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.577396 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.577406 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.577580 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.577618 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:58:35.577694 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.574117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:52235 (local address 127.8.13.193:40377)
0925 14:58:35.574179 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.574181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.574191 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:35.574234 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.574235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.574245 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.574295 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:35.574463 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.575216 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.576762 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.577321 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.577396 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.577406 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.577580 (+ 174us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.577618 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:58:35.577732 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.574043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:40377 (local address 127.8.13.194:52235)
0925 14:58:35.574100 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.574117 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:35.574171 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:35.574313 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:35.574314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:35.574317 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:35.574397 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.574400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.575298 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.575299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.576666 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.576669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.577440 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.577441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.577515 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.577518 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:35.577565 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:35.577627 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":63}
I20260925 14:58:35.577741 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.574172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39795 (local address 127.8.13.194:34157)
0925 14:58:35.574355 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.574363 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:35.574414 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:35.574548 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:35.574548 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:35.574549 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:35.574591 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.574592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.575395 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.575395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.576671 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:35.576673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.577419 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.577421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:35.577501 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.577507 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:58:35.577565 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 14:58:35.577622 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":28,"thread_start_us":104,"threads_started":1}
I20260925 14:58:35.578059 15281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a073adff209463facc7abcc15743c59" candidate_uuid: "f854b76c9c734857bbf73a2741070116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" is_pre_election: true
I20260925 14:58:35.578068 15554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a073adff209463facc7abcc15743c59" candidate_uuid: "f854b76c9c734857bbf73a2741070116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ca4a67ba77482c9c2c363f12d3055e" is_pre_election: true
I20260925 14:58:35.578155 15281 raft_consensus.cc:2454] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f854b76c9c734857bbf73a2741070116 in term 0.
I20260925 14:58:35.578191 15554 raft_consensus.cc:2454] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f854b76c9c734857bbf73a2741070116 in term 0.
I20260925 14:58:35.578341 15354 leader_election.cc:302] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed9f8f145a504cea8ac76a5d36f2dab3, f854b76c9c734857bbf73a2741070116; no voters:
I20260925 14:58:35.578477 15709 raft_consensus.cc:2781] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:35.578522 15709 raft_consensus.cc:482] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:35.578539 15709 raft_consensus.cc:3037] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:35.579218 15709 raft_consensus.cc:504] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.579329 15709 leader_election.cc:288] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [CANDIDATE]: Term 1 election: Requested vote from peers ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377), 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795)
I20260925 14:58:35.579489 15281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a073adff209463facc7abcc15743c59" candidate_uuid: "f854b76c9c734857bbf73a2741070116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3"
I20260925 14:58:35.579510 15554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a073adff209463facc7abcc15743c59" candidate_uuid: "f854b76c9c734857bbf73a2741070116" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59ca4a67ba77482c9c2c363f12d3055e"
I20260925 14:58:35.579586 15281 raft_consensus.cc:3037] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:35.579593 15554 raft_consensus.cc:3037] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:35.580307 15281 raft_consensus.cc:2454] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f854b76c9c734857bbf73a2741070116 in term 1.
I20260925 14:58:35.580307 15554 raft_consensus.cc:2454] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f854b76c9c734857bbf73a2741070116 in term 1.
I20260925 14:58:35.580471 15354 leader_election.cc:302] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59ca4a67ba77482c9c2c363f12d3055e, f854b76c9c734857bbf73a2741070116; no voters:
I20260925 14:58:35.580559 15709 raft_consensus.cc:2781] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:35.580688 15709 raft_consensus.cc:686] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 1 LEADER]: Becoming Leader. State: Replica: f854b76c9c734857bbf73a2741070116, State: Running, Role: LEADER
I20260925 14:58:35.580749 15709 consensus_queue.cc:237] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } }
I20260925 14:58:35.581111 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.580890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:56885)
0925 14:58:35.581048 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.581080 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143}
I20260925 14:58:35.581444 14961 catalog_manager.cc:5495] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 reported cstate change: term changed from 0 to 1, leader changed from <none> to f854b76c9c734857bbf73a2741070116 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "f854b76c9c734857bbf73a2741070116" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f854b76c9c734857bbf73a2741070116" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:35.646104 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.643868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49296 (local address 127.8.13.194:45079)
0925 14:58:35.643930 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:35.643932 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:35.643940 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:35.643977 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:35.643978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:35.644000 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:35.644058 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:35.644187 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.644653 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.645384 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:35.645578 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:35.645632 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:35.645788 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:35.645893 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:35.645954 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:35.646000 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 14:58:35.646023 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:35.647360 15281 raft_consensus.cc:1261] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Refusing update from remote peer f854b76c9c734857bbf73a2741070116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:35.647416 15554 raft_consensus.cc:1261] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Refusing update from remote peer f854b76c9c734857bbf73a2741070116: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:35.647671 15709 consensus_queue.cc:1035] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed9f8f145a504cea8ac76a5d36f2dab3" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 40377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:35.647794 15713 consensus_queue.cc:1035] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59ca4a67ba77482c9c2c363f12d3055e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:35.648430 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.648322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:43081)
0925 14:58:35.648362 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.648393 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 14:58:35.648597 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.648497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:60437)
0925 14:58:35.648540 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.648566 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 14:58:35.649514 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.649432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.193:43153)
0925 14:58:35.649467 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.649489 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:58:35.650507 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.650103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:36513)
0925 14:58:35.650449 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.650478 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331}
I20260925 14:58:35.650883 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.650797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.195:47297)
0925 14:58:35.650832 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.650854 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:58:35.650919 15719 mvcc.cc:205] Tried to move back new op lower bound from 7333266700889206784 to 7333266700619513856. Current Snapshot: MvccSnapshot[applied={T|T < 7333266700889206784}]
I20260925 14:58:35.651355 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.651262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39483 (local address 127.8.13.194:58787)
0925 14:58:35.651299 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.651331 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:39483: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260925 14:58:35.651813 14961 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:35.651904 14961 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36682:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:35.653164 14961 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 153aff0e44394f4a9f68cfb489c275ab: Sending DeleteTablet for 3 replicas of tablet 0a073adff209463facc7abcc15743c59
I20260925 14:58:35.653445 15398 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a073adff209463facc7abcc15743c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:35 UTC) from {username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:49290
I20260925 14:58:35.653517 15534 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a073adff209463facc7abcc15743c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:35 UTC) from {username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:39690
I20260925 14:58:35.653616 15261 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a073adff209463facc7abcc15743c59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:35 UTC) from {username='oryx', principal='oryx/127.8.13.253@KRBTEST.COM'} at 127.0.0.1:44866
I20260925 14:58:35.653707 15727 tablet_replica.cc:330] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: stopping tablet replica
I20260925 14:58:35.653756 15727 raft_consensus.cc:2229] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:35.653796 15727 raft_consensus.cc:2258] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:35.653806 15728 tablet_replica.cc:330] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3: stopping tablet replica
I20260925 14:58:35.653857 15728 raft_consensus.cc:2229] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:35.653898 15728 raft_consensus.cc:2258] T 0a073adff209463facc7abcc15743c59 P ed9f8f145a504cea8ac76a5d36f2dab3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:35.654129 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15109
I20260925 14:58:35.659322 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.659215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:40377 (local address 127.0.0.1:44874)
0925 14:58:35.659260 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.659289 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:40377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 14:58:35.659396 15726 tablet_replica.cc:330] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: stopping tablet replica
I20260925 14:58:35.659488 15726 raft_consensus.cc:2229] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 1 LEADER]: Raft consensus shutting down.
W20260925 14:58:35.659497 14948 catalog_manager.cc:4584] TS ed9f8f145a504cea8ac76a5d36f2dab3 (127.8.13.193:40377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a073adff209463facc7abcc15743c59: Network error: Client connection negotiation failed: client connection to 127.8.13.193:40377: connect: Connection refused (error 111)
I20260925 14:58:35.659610 15726 raft_consensus.cc:2258] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:35.659871 15726 ts_tablet_manager.cc:1905] T 0a073adff209463facc7abcc15743c59 P f854b76c9c734857bbf73a2741070116: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:35.660983 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15335
I20260925 14:58:35.663800 15727 ts_tablet_manager.cc:1905] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:35.665341 15727 ts_tablet_manager.cc:1918] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:35.665405 15727 log.cc:1196] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/0a073adff209463facc7abcc15743c59
I20260925 14:58:35.665613 15727 ts_tablet_manager.cc:1939] T 0a073adff209463facc7abcc15743c59 P 59ca4a67ba77482c9c2c363f12d3055e: Deleting consensus metadata
I20260925 14:58:35.666046 14948 catalog_manager.cc:4841] TS 59ca4a67ba77482c9c2c363f12d3055e (127.8.13.195:39795): tablet 0a073adff209463facc7abcc15743c59 (table test-table [id=d38f136c5fa54db48517d1dea44b8734]) successfully deleted
I20260925 14:58:35.667089 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:35.666987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45079 (local address 127.0.0.1:49306)
0925 14:58:35.667037 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:35.667061 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:45079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:58:35.667171 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15471
I20260925 14:58:35.674584 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 14827
I20260925 14:58:35.678758 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15015
2026-09-25T14:58:35Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6514 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:35 dist-test-slave-15hz krb5kdc[15731](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:35 dist-test-slave-15hz krb5kdc[15731](info): set up 2 sockets
Sep 25 14:58:35 dist-test-slave-15hz krb5kdc[15731](info): commencing operation
krb5kdc: starting...
W20260925 14:58:36.635728 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.937s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:36 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:36Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab.
Entry for principal oryx/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab.
I20260925 14:58:36.748628 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33829
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab
--principal=oryx/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:33829
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:36.827175 15744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:36.827378 15744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:36.827397 15744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:36.828622 15744 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:36.828665 15744 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260925 14:58:36.828676 15744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:36.828687 15744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:36.828698 15744 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:36.828709 15744 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:36.830150 15744 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:33829
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.254.keytab
--principal=oryx/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:36.830341 15744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:36.830518 15744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:36.835055 15749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.835057 15752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:36.835167 15750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:36.835091 15744 server_base.cc:1034] running on GCE node
I20260925 14:58:36.835525 15744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:36.835847 15744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:36.837020 15744 hybrid_clock.cc:648] HybridClock initialized: now 1790348316837003 us; error 32 us; skew 500 ppm
Sep 25 14:58:36 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:36.839521 15744 init.cc:377] Logged in from keytab as oryx/127.8.13.254@KRBTEST.COM (short username oryx)
I20260925 14:58:36.840035 15744 webserver.cc:460] Webserver started at http://127.8.13.254:42413/ using document root <none> and password file <none>
I20260925 14:58:36.840242 15744 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:36.840294 15744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:36.840428 15744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:36.841553 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "5248a29df19e4193b0567b1bb5477836"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-15hz"
server_key: "baa76c0e6aaded68a2879136d4171234"
server_key_iv: "9b95cb312a315a54a445f9f39c207c72"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.841914 15744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "5248a29df19e4193b0567b1bb5477836"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-15hz"
server_key: "baa76c0e6aaded68a2879136d4171234"
server_key_iv: "9b95cb312a315a54a445f9f39c207c72"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.843318 15744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:36.844108 15759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.844282 15744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:36.844373 15744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "5248a29df19e4193b0567b1bb5477836"
format_stamp: "Formatted at 2026-09-25 14:58:36 on dist-test-slave-15hz"
server_key: "baa76c0e6aaded68a2879136d4171234"
server_key_iv: "9b95cb312a315a54a445f9f39c207c72"
server_key_version: "encryptionkey@0"
I20260925 14:58:36.844451 15744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:36.859122 15744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:36.861037 15744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:36.861187 15744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:36.865831 15744 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:33829
I20260925 14:58:36.865895 15821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:33829 every 8 connection(s)
I20260925 14:58:36.866207 15744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:36.866846 15822 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:36.869102 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Bootstrap starting.
I20260925 14:58:36.869644 15822 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:36.869868 15822 log.cc:824] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:36.870478 15822 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: No bootstrap required, opened a new log
I20260925 14:58:36.872501 15822 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } }
I20260925 14:58:36.872640 15822 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:36.872660 15822 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5248a29df19e4193b0567b1bb5477836, State: Initialized, Role: FOLLOWER
I20260925 14:58:36.872771 15822 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } }
I20260925 14:58:36.872843 15822 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:36.872870 15822 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:36.872902 15822 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:36.873371 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15744
I20260925 14:58:36.873373 15822 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } }
I20260925 14:58:36.873457 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:36.873450 15822 leader_election.cc:302] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5248a29df19e4193b0567b1bb5477836; no voters:
I20260925 14:58:36.873607 15822 leader_election.cc:288] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:36.873623 8247 external_mini_cluster.cc:1428] Setting key 908d46244087c74288adbb1cfe3d381e
I20260925 14:58:36.873658 15825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:36.873802 15825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [term 1 LEADER]: Becoming Leader. State: Replica: 5248a29df19e4193b0567b1bb5477836, State: Running, Role: LEADER
I20260925 14:58:36.873868 15822 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:36.873966 15825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } }
I20260925 14:58:36.874297 15825 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5248a29df19e4193b0567b1bb5477836. Latest consensus state: current_term: 1 leader_uuid: "5248a29df19e4193b0567b1bb5477836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } } }
I20260925 14:58:36.874310 15827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5248a29df19e4193b0567b1bb5477836" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5248a29df19e4193b0567b1bb5477836" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33829 } } }
I20260925 14:58:36.874503 15827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:36.874470 15825 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:36.875138 15836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:36.876080 15836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:36.877967 15836 catalog_manager.cc:1347] Generated new cluster ID: 1ff92ad6dad54859a0c6f7e645e01837
I20260925 14:58:36.878022 15836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:36 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:36.885000 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:36.874325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.8.13.254:33829)
0925 14:58:36.874559 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:36.874570 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:36.874605 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:58:36.874721 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:36.874727 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:36.874773 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:36.874908 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:36.875619 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.876917 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.877911 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:36.878508 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:36.878608 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:36.881531 (+ 2923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:36.881536 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:36.881536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:36.881548 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:36.883354 (+ 1806us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.883772 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.883775 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.883776 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.883879 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:36.884068 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:36.884069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:36.884069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:36.884360 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:36.884503 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:36.884604 (+ 101us) server_negotiation.cc:299] Negotiation successful
0925 14:58:36.884750 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":76,"threads_started":1}
I20260925 14:58:36.898072 15836 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:36.898608 15836 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:36.907650 15836 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Generated new TSK 0
I20260925 14:58:36.907827 15836 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
I20260925 14:58:36.934219 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33829
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:37.012701 15848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:37.012912 15848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:37.012944 15848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:37.014132 15848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:37.014175 15848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:37.014204 15848 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:37.014226 15848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:37.015779 15848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:37.016029 15848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:37.016248 15848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:37.021031 15853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.021040 15854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.021031 15856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:37.021333 15848 server_base.cc:1034] running on GCE node
I20260925 14:58:37.021541 15848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:37.021759 15848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:37.022919 15848 hybrid_clock.cc:648] HybridClock initialized: now 1790348317022905 us; error 25 us; skew 500 ppm
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:37.025458 15848 init.cc:377] Logged in from keytab as oryx/127.8.13.193@KRBTEST.COM (short username oryx)
I20260925 14:58:37.025908 15848 webserver.cc:460] Webserver started at http://127.8.13.193:37971/ using document root <none> and password file <none>
I20260925 14:58:37.026135 15848 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:37.026180 15848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:37.026306 15848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:37.027351 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "d7ff56526a3c4594a92f4777fb7ed204"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "be05e48d8416c3317f53b1a2facd8673"
server_key_iv: "187133eeff01f5196a5325a9e8878933"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.027702 15848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "d7ff56526a3c4594a92f4777fb7ed204"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "be05e48d8416c3317f53b1a2facd8673"
server_key_iv: "187133eeff01f5196a5325a9e8878933"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.028923 15848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:37.029599 15863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.029767 15848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:37.029841 15848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "d7ff56526a3c4594a92f4777fb7ed204"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "be05e48d8416c3317f53b1a2facd8673"
server_key_iv: "187133eeff01f5196a5325a9e8878933"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.029894 15848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:37.073794 15848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:37.075709 15848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:37.075812 15848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:37.076258 15848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:37.076294 15848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.076315 15848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:37.076333 15848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.082562 15848 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:42717
I20260925 14:58:37.082645 15978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:42717 every 8 connection(s)
I20260925 14:58:37.082935 15848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:37.085479 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.076781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:40749 (local address 127.8.13.254:33829)
0925 14:58:37.076845 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.076847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.077032 (+ 185us) server_negotiation.cc:407] Connection header received
0925 14:58:37.077589 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.077591 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.077608 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.077688 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.078574 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.079323 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.080408 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.080630 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.080687 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.082826 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.082827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.082828 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.082836 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.084178 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.084541 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.084542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.084542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.084593 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.084817 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.084817 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.084818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.084926 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.085032 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.085290 (+ 258us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.085387 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 14:58:37.085633 15872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.076736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.193:40749)
0925 14:58:37.076902 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.076944 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.077481 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.077781 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.077785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.078186 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.078466 (+ 280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.078474 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.079416 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.079417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.080319 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.080322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.080712 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.080713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.080835 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.081359 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.081367 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.082739 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.084251 (+ 1512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.084257 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.084259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.084475 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.084646 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.084647 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.084647 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.084762 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.085033 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.085040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.085173 (+ 133us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.085260 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.085414 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260925 14:58:37.086066 15979 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33829
I20260925 14:58:37.086144 15979 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:37.086324 15979 heartbeater.cc:511] Master 127.8.13.254:33829 requested a full tablet report, sending...
I20260925 14:58:37.086895 15775 ts_manager.cc:194] Registered new tserver with Master: d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717)
I20260925 14:58:37.087579 15775 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:40749
I20260925 14:58:37.089455 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15848
I20260925 14:58:37.089564 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:37.089735 8247 external_mini_cluster.cc:1428] Setting key 942fcea7ae3ce91b55799b88d0e7ac59
I20260925 14:58:37.092435 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.087061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:51189 (local address 127.8.13.254:33829)
0925 14:58:37.087094 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.087096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.087205 (+ 109us) server_negotiation.cc:407] Connection header received
0925 14:58:37.087256 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.087257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.087267 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.087310 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.087621 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.088320 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.088992 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.089225 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.089314 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.090122 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.090123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.090123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.090129 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.091477 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.091795 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.091796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.091796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.091850 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.092000 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.092001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.092002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.092101 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.092161 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.092277 (+ 116us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.092349 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260925 14:58:37.092447 15985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.087028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.193:51189)
0925 14:58:37.087142 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.087152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.087205 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.087312 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.087313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.087483 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.087566 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.087569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.088408 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.088408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.088922 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.088924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.089307 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.089307 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.089350 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.089725 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.089730 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.090063 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.091567 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.091568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.091568 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.091724 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.091902 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.091903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.091903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.091952 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.092154 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.092154 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.092199 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.092250 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.092347 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for oryx/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
I20260925 14:58:37.112286 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33829
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:37.190989 15988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:37.191200 15988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:37.191224 15988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:37.192380 15988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:37.192417 15988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:37.192436 15988 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:37.192451 15988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:37.193904 15988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:37.194087 15988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:37.194262 15988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:37.198840 15993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.198872 15994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.198915 15996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:37.199018 15988 server_base.cc:1034] running on GCE node
I20260925 14:58:37.199219 15988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:37.199426 15988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:37.200582 15988 hybrid_clock.cc:648] HybridClock initialized: now 1790348317200557 us; error 30 us; skew 500 ppm
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:37.202889 15988 init.cc:377] Logged in from keytab as oryx/127.8.13.194@KRBTEST.COM (short username oryx)
I20260925 14:58:37.203305 15988 webserver.cc:460] Webserver started at http://127.8.13.194:42403/ using document root <none> and password file <none>
I20260925 14:58:37.203480 15988 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:37.203518 15988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:37.203660 15988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:37.204671 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "a2c91ce8de2eddc5d85c7096072d2815"
server_key_iv: "5a484ca4e663b2a5aaa0ada8048b77b6"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.204986 15988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "a2c91ce8de2eddc5d85c7096072d2815"
server_key_iv: "5a484ca4e663b2a5aaa0ada8048b77b6"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.206280 15988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:37.206988 16003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.207172 15988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:37.207252 15988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "a2c91ce8de2eddc5d85c7096072d2815"
server_key_iv: "5a484ca4e663b2a5aaa0ada8048b77b6"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.207309 15988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:37.215876 15988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:37.217754 15988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:37.217859 15988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:37.218310 15988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:37.218351 15988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.218374 15988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:37.218384 15988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.223928 15988 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:39955
I20260925 14:58:37.224010 16118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:39955 every 8 connection(s)
I20260925 14:58:37.224321 15988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.226617 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 15988
I20260925 14:58:37.226718 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:37.226914 8247 external_mini_cluster.cc:1428] Setting key 88e336c2f404f7eff2765abc2d07023f
I20260925 14:58:37.227710 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.218776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56279 (local address 127.8.13.254:33829)
0925 14:58:37.218820 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.218822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.219037 (+ 215us) server_negotiation.cc:407] Connection header received
0925 14:58:37.219570 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.219572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.219584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.219634 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.220404 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.221160 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.222540 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.222824 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.222894 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.225352 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.225353 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.225353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.225360 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.226404 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.226775 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.226777 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.226777 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.226836 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.227071 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.227073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.227073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.227214 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.227273 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.227520 (+ 247us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.227629 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:37.227892 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.218782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.194:56279)
0925 14:58:37.218927 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.218962 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.219447 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.219705 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.219709 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.220038 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.220319 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.220327 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.221240 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.221241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.222454 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.222457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.222922 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.222924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.223094 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.223665 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.223674 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.225268 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.226470 (+ 1202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.226476 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.226478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.226691 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.226891 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.226892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.226892 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.227014 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.227264 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.227270 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.227387 (+ 117us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.227490 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.227676 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260925 14:58:37.229326 16119 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33829
I20260925 14:58:37.229415 16119 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:37.229604 16119 heartbeater.cc:511] Master 127.8.13.254:33829 requested a full tablet report, sending...
I20260925 14:58:37.229991 15776 ts_manager.cc:194] Registered new tserver with Master: 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955)
I20260925 14:58:37.230434 15776 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:56279
WARNING: no policy specified for oryx/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 14:58:37.236027 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.230546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:40237 (local address 127.8.13.254:33829)
0925 14:58:37.230580 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.230581 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.230597 (+ 16us) server_negotiation.cc:407] Connection header received
0925 14:58:37.230615 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.230616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.230625 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.230681 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.231042 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.231604 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.232590 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.232772 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.232816 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.233764 (+ 948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.233765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.233766 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.233772 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.234984 (+ 1212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.235283 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.235284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.235285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.235336 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.235494 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.235495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.235495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.235666 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.235724 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.235901 (+ 177us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.235964 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:58:37.236037 16126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.230202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.194:40237)
0925 14:58:37.230325 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.230336 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.230383 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.230699 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.230700 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.230899 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.230965 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.230967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.231705 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.231706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.232514 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.232517 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.232835 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.232836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.232899 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.233302 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.233307 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.233691 (+ 384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.235059 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.235060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.235060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.235224 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.235387 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.235387 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.235388 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.235443 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.235722 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.235723 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.235800 (+ 77us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.235862 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.235961 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
I20260925 14:58:37.250507 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33829
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45981
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:37.331312 16128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:37.331555 16128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:37.331590 16128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:37.332782 16128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:37.332823 16128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:37.332844 16128 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260925 14:58:37.332883 16128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:37.334487 16128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45981
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.13.254:33829
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:37.334698 16128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:37.334931 16128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:37.339735 16133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:37.339834 16128 server_base.cc:1034] running on GCE node
W20260925 14:58:37.339896 16134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:37.339761 16136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:37.340184 16128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:37.340456 16128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:37.341605 16128 hybrid_clock.cc:648] HybridClock initialized: now 1790348317341580 us; error 38 us; skew 500 ppm
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:37.344399 16128 init.cc:377] Logged in from keytab as oryx/127.8.13.195@KRBTEST.COM (short username oryx)
I20260925 14:58:37.344878 16128 webserver.cc:460] Webserver started at http://127.8.13.195:39157/ using document root <none> and password file <none>
I20260925 14:58:37.345085 16128 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:37.345127 16128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:37.345248 16128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:37.346289 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "46e410a3589e4edd91900c726c44d29d"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "4da03bd26274f35e70f767024e0ef5ae"
server_key_iv: "19d74850e48a87810a2aacff9771abba"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.346609 16128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "46e410a3589e4edd91900c726c44d29d"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "4da03bd26274f35e70f767024e0ef5ae"
server_key_iv: "19d74850e48a87810a2aacff9771abba"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.347944 16128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:37.348665 16143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.348804 16128 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.348878 16128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "46e410a3589e4edd91900c726c44d29d"
format_stamp: "Formatted at 2026-09-25 14:58:37 on dist-test-slave-15hz"
server_key: "4da03bd26274f35e70f767024e0ef5ae"
server_key_iv: "19d74850e48a87810a2aacff9771abba"
server_key_version: "encryptionkey@0"
I20260925 14:58:37.348946 16128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:37.365345 16128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:37.367434 16128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:37.367586 16128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:37.368093 16128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:37.368131 16128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.368155 16128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:37.368166 16128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.374498 16128 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:40805
I20260925 14:58:37.374603 16258 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:40805 every 8 connection(s)
I20260925 14:58:37.375022 16128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:37.375274 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16128
I20260925 14:58:37.375383 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:37.375615 8247 external_mini_cluster.cc:1428] Setting key 678a11f8485ed9745add4d286424df84
I20260925 14:58:37.377332 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.368476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:45001 (local address 127.8.13.254:33829)
0925 14:58:37.368537 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.368540 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.368710 (+ 170us) server_negotiation.cc:407] Connection header received
0925 14:58:37.369185 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.369187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.369200 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.369267 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.370031 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.370820 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.372098 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.372285 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.372341 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.374567 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.374569 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.374569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.374577 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.376234 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.376652 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.376653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.376653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.376716 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.376878 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.376878 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.376879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.376976 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.377017 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.377210 (+ 193us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.377272 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:37.377503 16152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.368442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.195:45001)
0925 14:58:37.368583 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.368629 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.369093 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.369339 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.369343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.369654 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.369933 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.369939 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.370927 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.370928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.372002 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.372005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.372365 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.372366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.372489 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.372948 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.372955 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.374467 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.376310 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.376315 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.376317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.376595 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.376757 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.376758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.376758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.376844 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.377007 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.377009 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.377120 (+ 111us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.377190 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.377306 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260925 14:58:37.377909 16259 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33829
I20260925 14:58:37.377981 16259 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:37.378167 16259 heartbeater.cc:511] Master 127.8.13.254:33829 requested a full tablet report, sending...
I20260925 14:58:37.378531 15776 ts_manager.cc:194] Registered new tserver with Master: 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.379009 15776 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:45001
I20260925 14:58:37.383956 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.378881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:39263 (local address 127.8.13.254:33829)
0925 14:58:37.378917 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.378919 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.379007 (+ 88us) server_negotiation.cc:407] Connection header received
0925 14:58:37.379042 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.379043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.379052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.379110 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.379499 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.380098 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.380813 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.380986 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.381031 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.381888 (+ 857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.381890 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.381890 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.381896 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.383070 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.383364 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.383364 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.383365 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.383409 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.383564 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.383565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.383565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.383681 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.383729 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.383838 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.383893 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 14:58:37.383972 16265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.378801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.8.13.195:39263)
0925 14:58:37.378946 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.378955 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.379004 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.379155 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.379156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.379369 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.379428 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.379430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.380202 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.380203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.380747 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.380748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.381039 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.381039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.381088 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.381509 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.381514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.381809 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.383145 (+ 1336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.383147 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.383147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.383308 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.383444 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.383444 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.383444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.383493 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.383726 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.383727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.383765 (+ 38us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.383816 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.383892 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":54,"threads_started":1}
I20260925 14:58:37.387105 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:58:37.406448 15833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:37.401379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.8.13.254:33829)
0925 14:58:37.401439 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.401441 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.401449 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:37.401477 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.401478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.401489 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.401534 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.402076 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.402554 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.403290 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.403488 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.403578 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.406363 (+ 2785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43658: BlockingRecv error: recv got EOF from 127.0.0.1:43658 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.413506 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.407154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43664 (local address 127.8.13.254:33829)
0925 14:58:37.407202 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.407205 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.407236 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:58:37.407306 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.407307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.407317 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.407360 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.407912 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.408385 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.409046 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.409227 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.409280 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.411403 (+ 2123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.411404 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.411405 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.411412 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.412529 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.412832 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.412833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.412833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.412885 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.413043 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.413044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.413044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.413159 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.413243 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.413340 (+ 97us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.413416 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:37.414805 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:37.416437 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:37.422442 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:42717 (local address 127.0.0.1:50088)
0925 14:58:37.418843 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.418875 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.418946 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.419223 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.419223 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.419226 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.419333 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.419334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420272 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.420273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.421469 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.421472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.422133 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.422133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.422216 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.422229 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.422304 (+ 75us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.422359 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":41,"thread_start_us":75,"threads_started":1}
I20260925 14:58:37.422453 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50088 (local address 127.8.13.193:42717)
0925 14:58:37.418794 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.418797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.418940 (+ 143us) server_negotiation.cc:407] Connection header received
0925 14:58:37.419049 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.419051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.419068 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.419121 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.419424 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420196 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.421551 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.422054 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.422118 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.422209 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.422321 (+ 112us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.422360 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":74,"threads_started":1}
I20260925 14:58:37.422842 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.0.0.1:46580)
0925 14:58:37.418843 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.418880 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.418947 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.419190 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.419192 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.419194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.419307 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.419309 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420635 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.420636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.421642 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.421644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.422618 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.422619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.422687 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.422691 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.422734 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.422770 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":23,"thread_start_us":65,"threads_started":1}
I20260925 14:58:37.423079 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46580 (local address 127.8.13.194:39955)
0925 14:58:37.418770 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.418775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.419030 (+ 255us) server_negotiation.cc:407] Connection header received
0925 14:58:37.419073 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.419076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.419112 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.419177 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.419371 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420551 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.421709 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.422445 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.422511 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.422582 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.422950 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.422989 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":85,"threads_started":1}
I20260925 14:58:37.423187 15913 tablet_service.cc:1467] Processing CreateTablet for tablet 83c855f768794ea2b08170c04ee60e2a (DEFAULT_TABLE table=test-table [id=5e1b90346625407881317756928004e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.423605 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c855f768794ea2b08170c04ee60e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.423699 16053 tablet_service.cc:1467] Processing CreateTablet for tablet 83c855f768794ea2b08170c04ee60e2a (DEFAULT_TABLE table=test-table [id=5e1b90346625407881317756928004e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.424021 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c855f768794ea2b08170c04ee60e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.424273 16285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40805 (local address 127.0.0.1:33950)
0925 14:58:37.418990 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.418999 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.419039 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.419239 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.419240 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.419241 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.419444 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.419447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.421714 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.421714 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.423061 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.423064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.424057 (+ 993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.424058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.424114 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.424118 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.424167 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.424207 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":154,"thread_start_us":62,"threads_started":1}
I20260925 14:58:37.424537 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.418837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33950 (local address 127.8.13.195:40805)
0925 14:58:37.419022 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.419025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.419034 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:37.419100 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.419102 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.419115 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.419179 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.419515 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.420550 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.423148 (+ 2598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.423887 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.423954 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.424022 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.424412 (+ 390us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.424450 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":76,"threads_started":1}
I20260925 14:58:37.425170 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 83c855f768794ea2b08170c04ee60e2a (DEFAULT_TABLE table=test-table [id=5e1b90346625407881317756928004e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.425498 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83c855f768794ea2b08170c04ee60e2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.426965 16287 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Bootstrap starting.
I20260925 14:58:37.427594 16287 tablet_bootstrap.cc:654] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.427624 16288 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Bootstrap starting.
I20260925 14:58:37.428009 16287 log.cc:824] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:37.428216 16288 tablet_bootstrap.cc:654] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.428475 16288 log.cc:824] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:37.428834 16287 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: No bootstrap required, opened a new log
I20260925 14:58:37.428826 16291 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Bootstrap starting.
I20260925 14:58:37.428907 16287 ts_tablet_manager.cc:1397] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.429316 16288 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: No bootstrap required, opened a new log
I20260925 14:58:37.429375 16288 ts_tablet_manager.cc:1397] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.429443 16291 tablet_bootstrap.cc:654] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.429749 16291 log.cc:824] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:37.430505 16291 tablet_bootstrap.cc:492] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: No bootstrap required, opened a new log
I20260925 14:58:37.430576 16291 ts_tablet_manager.cc:1397] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.430837 16287 raft_consensus.cc:348] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.431010 16287 raft_consensus.cc:374] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.431042 16287 raft_consensus.cc:729] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.431008 16288 raft_consensus.cc:348] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.431161 16287 consensus_queue.cc:260] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.431207 16288 raft_consensus.cc:374] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.431250 16288 raft_consensus.cc:729] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.431372 16288 consensus_queue.cc:260] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.431409 16287 ts_tablet_manager.cc:1428] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:37.431555 15979 heartbeater.cc:503] Master 127.8.13.254:33829 was elected leader, sending a full tablet report...
I20260925 14:58:37.431950 16119 heartbeater.cc:503] Master 127.8.13.254:33829 was elected leader, sending a full tablet report...
I20260925 14:58:37.432173 16288 ts_tablet_manager.cc:1428] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 14:58:37.432514 16291 raft_consensus.cc:348] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.432669 16291 raft_consensus.cc:374] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.432713 16291 raft_consensus.cc:729] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.432817 16291 consensus_queue.cc:260] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.433071 16291 ts_tablet_manager.cc:1428] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:37.433118 16259 heartbeater.cc:503] Master 127.8.13.254:33829 was elected leader, sending a full tablet report...
W20260925 14:58:37.474993 16120 tablet.cc:2369] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:37.583817 15980 tablet.cc:2369] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:37.625890 16260 tablet.cc:2369] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:37.639104 16293 raft_consensus.cc:482] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:37.639226 16293 raft_consensus.cc:504] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.639513 16293 leader_election.cc:288] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.642338 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.639807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:46783 (local address 127.8.13.194:39955)
0925 14:58:37.639860 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.639861 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.639924 (+ 63us) server_negotiation.cc:407] Connection header received
0925 14:58:37.639996 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.639997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.640009 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.640076 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.640230 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.640699 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.641683 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.642034 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.642093 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.642100 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.642230 (+ 130us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.642269 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:37.642334 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.639996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:38299 (local address 127.8.13.195:40805)
0925 14:58:37.640048 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.640051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.640059 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:37.640123 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.640124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.640135 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.640179 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.640296 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.640753 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.641682 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.642030 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.642088 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.642097 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.642221 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.642258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:58:37.642354 16297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.639903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40805 (local address 127.8.13.193:38299)
0925 14:58:37.640009 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.640017 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.640056 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.640187 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.640187 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.640190 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.640244 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.640245 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.640817 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.640818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.641600 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.641602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.642101 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.642102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.642165 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.642168 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.642211 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.642258 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":16,"thread_start_us":49,"threads_started":1}
I20260925 14:58:37.642362 16296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.639709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.8.13.193:46783)
0925 14:58:37.639845 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.639857 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.639924 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.640106 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.640107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.640110 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.640172 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.640175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.640770 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.640771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.641599 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.641602 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.642104 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.642105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.642173 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.642176 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.642220 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.642266 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":1,"thread_start_us":79,"threads_started":1}
I20260925 14:58:37.642752 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c855f768794ea2b08170c04ee60e2a" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d" is_pre_election: true
I20260925 14:58:37.642760 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c855f768794ea2b08170c04ee60e2a" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" is_pre_election: true
I20260925 14:58:37.642889 16213 raft_consensus.cc:2454] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 0.
I20260925 14:58:37.642889 16073 raft_consensus.cc:2454] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 0.
I20260925 14:58:37.643121 15867 leader_election.cc:302] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e410a3589e4edd91900c726c44d29d, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:37.643255 16293 raft_consensus.cc:2781] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:37.643306 16293 raft_consensus.cc:482] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:37.643319 16293 raft_consensus.cc:3037] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.644028 16293 raft_consensus.cc:504] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.644140 16293 leader_election.cc:288] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.644312 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c855f768794ea2b08170c04ee60e2a" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
I20260925 14:58:37.644354 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83c855f768794ea2b08170c04ee60e2a" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d"
I20260925 14:58:37.644418 16213 raft_consensus.cc:3037] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.644407 16073 raft_consensus.cc:3037] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.645300 16213 raft_consensus.cc:2454] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 1.
I20260925 14:58:37.645300 16073 raft_consensus.cc:2454] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 1.
I20260925 14:58:37.645450 15867 leader_election.cc:302] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:37.645569 16293 raft_consensus.cc:2781] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:37.645678 16293 raft_consensus.cc:686] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 1 LEADER]: Becoming Leader. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Running, Role: LEADER
I20260925 14:58:37.645774 16293 consensus_queue.cc:237] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.646515 15775 catalog_manager.cc:5495] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "d7ff56526a3c4594a92f4777fb7ed204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:37.706084 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.704090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50102 (local address 127.8.13.193:42717)
0925 14:58:37.704135 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.704137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.704147 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:37.704191 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.704192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.704213 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.704273 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:37.704366 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.704840 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.705520 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.705702 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.705743 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.705844 (+ 101us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.705941 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.705994 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.706010 (+ 16us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.706027 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:37.707851 16213 raft_consensus.cc:1261] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Refusing update from remote peer d7ff56526a3c4594a92f4777fb7ed204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.707851 16073 raft_consensus.cc:1261] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Refusing update from remote peer d7ff56526a3c4594a92f4777fb7ed204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.708168 16298 consensus_queue.cc:1035] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.708285 16293 consensus_queue.cc:1035] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.710457 16301 mvcc.cc:205] Tried to move back new op lower bound from 7333266709328769024 to 7333266709077712896. Current Snapshot: MvccSnapshot[applied={T|T < 7333266709328769024}]
I20260925 14:58:37.710584 16303 mvcc.cc:205] Tried to move back new op lower bound from 7333266709328769024 to 7333266709077712896. Current Snapshot: MvccSnapshot[applied={T|T < 7333266709328769024}]
I20260925 14:58:37.710775 16302 mvcc.cc:205] Tried to move back new op lower bound from 7333266709328769024 to 7333266709077712896. Current Snapshot: MvccSnapshot[applied={T|T < 7333266709328769024}]
I20260925 14:58:37.712896 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:37.712987 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:37.713675 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Sending DeleteTablet for 3 replicas of tablet 83c855f768794ea2b08170c04ee60e2a
I20260925 14:58:37.713976 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c855f768794ea2b08170c04ee60e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50088
I20260925 14:58:37.714057 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c855f768794ea2b08170c04ee60e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46580
I20260925 14:58:37.714262 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 83c855f768794ea2b08170c04ee60e2a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33950
I20260925 14:58:37.714363 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260925 14:58:37.714468 16311 tablet_replica.cc:330] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: stopping tablet replica
W20260925 14:58:37.714510 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:37.714553 16311 raft_consensus.cc:2229] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:37.714679 16311 raft_consensus.cc:2258] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.714826 16312 tablet_replica.cc:330] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: stopping tablet replica
I20260925 14:58:37.714963 16312 raft_consensus.cc:2229] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:37.715010 16313 tablet_replica.cc:330] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: stopping tablet replica
I20260925 14:58:37.715107 16313 raft_consensus.cc:2229] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:37.715183 16312 raft_consensus.cc:2258] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.715219 16313 raft_consensus.cc:2258] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.715754 16053 tablet_service.cc:1467] Processing CreateTablet for tablet d45a818d2ef54814bccf6161e58f5489 (DEFAULT_TABLE table=test-table [id=2bb0efb0c154494ca51911133243a684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.715790 16193 tablet_service.cc:1467] Processing CreateTablet for tablet d45a818d2ef54814bccf6161e58f5489 (DEFAULT_TABLE table=test-table [id=2bb0efb0c154494ca51911133243a684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.715859 15913 tablet_service.cc:1467] Processing CreateTablet for tablet d45a818d2ef54814bccf6161e58f5489 (DEFAULT_TABLE table=test-table [id=2bb0efb0c154494ca51911133243a684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.715878 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45a818d2ef54814bccf6161e58f5489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.715906 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45a818d2ef54814bccf6161e58f5489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.715945 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d45a818d2ef54814bccf6161e58f5489. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.717211 16287 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Bootstrap starting.
I20260925 14:58:37.717211 16288 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Bootstrap starting.
I20260925 14:58:37.717250 16291 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Bootstrap starting.
I20260925 14:58:37.717535 16311 ts_tablet_manager.cc:1905] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.717628 16288 tablet_bootstrap.cc:654] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.717629 16287 tablet_bootstrap.cc:654] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.717629 16291 tablet_bootstrap.cc:654] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.718400 16291 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: No bootstrap required, opened a new log
I20260925 14:58:37.718444 16291 ts_tablet_manager.cc:1397] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:37.718591 16291 raft_consensus.cc:348] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.718679 16291 raft_consensus.cc:374] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.718679 16287 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: No bootstrap required, opened a new log
I20260925 14:58:37.718701 16291 raft_consensus.cc:729] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.718713 16287 ts_tablet_manager.cc:1397] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.718743 16291 consensus_queue.cc:260] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.718825 16291 ts_tablet_manager.cc:1428] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.718842 16287 raft_consensus.cc:348] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.718897 16287 raft_consensus.cc:374] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.718916 16287 raft_consensus.cc:729] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.718954 16287 consensus_queue.cc:260] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.719022 16287 ts_tablet_manager.cc:1428] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.719096 16312 ts_tablet_manager.cc:1905] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.720029 16288 tablet_bootstrap.cc:492] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: No bootstrap required, opened a new log
I20260925 14:58:37.720079 16288 ts_tablet_manager.cc:1397] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:37.720216 16288 raft_consensus.cc:348] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.720279 16288 raft_consensus.cc:374] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.720296 16288 raft_consensus.cc:729] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.720342 16288 consensus_queue.cc:260] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.720425 16288 ts_tablet_manager.cc:1428] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:37.720602 16311 ts_tablet_manager.cc:1918] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:37.720681 16311 log.cc:1196] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/83c855f768794ea2b08170c04ee60e2a
I20260925 14:58:37.720757 16312 ts_tablet_manager.cc:1918] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:37.720875 16312 log.cc:1196] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/83c855f768794ea2b08170c04ee60e2a
I20260925 14:58:37.720927 16311 ts_tablet_manager.cc:1939] T 83c855f768794ea2b08170c04ee60e2a P d7ff56526a3c4594a92f4777fb7ed204: Deleting consensus metadata
I20260925 14:58:37.721112 16312 ts_tablet_manager.cc:1939] T 83c855f768794ea2b08170c04ee60e2a P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting consensus metadata
I20260925 14:58:37.721328 15762 catalog_manager.cc:4841] TS d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717): tablet 83c855f768794ea2b08170c04ee60e2a (table test-table [id=5e1b90346625407881317756928004e4]) successfully deleted
I20260925 14:58:37.721464 15763 catalog_manager.cc:4841] TS 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955): tablet 83c855f768794ea2b08170c04ee60e2a (table test-table [id=5e1b90346625407881317756928004e4]) successfully deleted
I20260925 14:58:37.721801 16313 ts_tablet_manager.cc:1905] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.723243 16313 ts_tablet_manager.cc:1918] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:37.723315 16313 log.cc:1196] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/83c855f768794ea2b08170c04ee60e2a
I20260925 14:58:37.723562 16313 ts_tablet_manager.cc:1939] T 83c855f768794ea2b08170c04ee60e2a P 46e410a3589e4edd91900c726c44d29d: Deleting consensus metadata
I20260925 14:58:37.724054 15763 catalog_manager.cc:4841] TS 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805): tablet 83c855f768794ea2b08170c04ee60e2a (table test-table [id=5e1b90346625407881317756928004e4]) successfully deleted
I20260925 14:58:37.747505 16306 raft_consensus.cc:482] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:37.747653 16306 raft_consensus.cc:504] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.747814 16306 leader_election.cc:288] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.748037 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45a818d2ef54814bccf6161e58f5489" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d" is_pre_election: true
I20260925 14:58:37.748134 16213 raft_consensus.cc:2454] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 0.
I20260925 14:58:37.748241 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45a818d2ef54814bccf6161e58f5489" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" is_pre_election: true
I20260925 14:58:37.748279 15867 leader_election.cc:302] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e410a3589e4edd91900c726c44d29d, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:37.748325 16073 raft_consensus.cc:2454] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 0.
I20260925 14:58:37.748396 16306 raft_consensus.cc:2781] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:37.748433 16306 raft_consensus.cc:482] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:37.748451 16306 raft_consensus.cc:3037] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.749112 16306 raft_consensus.cc:504] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.749228 16306 leader_election.cc:288] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 election: Requested vote from peers 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.749423 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45a818d2ef54814bccf6161e58f5489" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
I20260925 14:58:37.749500 16073 raft_consensus.cc:3037] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.750243 16073 raft_consensus.cc:2454] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 1.
I20260925 14:58:37.750353 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d45a818d2ef54814bccf6161e58f5489" candidate_uuid: "d7ff56526a3c4594a92f4777fb7ed204" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d"
I20260925 14:58:37.750394 15867 leader_election.cc:302] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:37.750427 16213 raft_consensus.cc:3037] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.750471 16306 raft_consensus.cc:2781] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:37.750511 16306 raft_consensus.cc:686] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 LEADER]: Becoming Leader. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Running, Role: LEADER
I20260925 14:58:37.750558 16306 consensus_queue.cc:237] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.751143 16213 raft_consensus.cc:2454] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7ff56526a3c4594a92f4777fb7ed204 in term 1.
I20260925 14:58:37.751350 15776 catalog_manager.cc:5495] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 reported cstate change: term changed from 0 to 1, leader changed from <none> to d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "d7ff56526a3c4594a92f4777fb7ed204" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } health_report { overall_health: UNKNOWN } } }
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348316, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.254@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.783393 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.776329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43668 (local address 127.8.13.254:33829)
0925 14:58:37.776381 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.776383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.776511 (+ 128us) server_negotiation.cc:407] Connection header received
0925 14:58:37.776582 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.776582 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.776591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.776640 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.777212 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.777697 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.778688 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.778867 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.778909 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.781118 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.781119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.781119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.781125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.782314 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.782734 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.782735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.782735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.782778 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.782959 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.782960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.782960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.783053 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.783097 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.783263 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.783328 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:58:37.783515 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.776275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33829 (local address 127.0.0.1:43668)
0925 14:58:37.776430 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.776442 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.776523 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.776659 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.776660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.777050 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:37.777137 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.777140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.777769 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.777770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.778613 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.778616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.778930 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.778932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.778988 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.779500 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:37.779505 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:37.781039 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:37.782393 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.782400 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.782402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.782665 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.782830 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:37.782831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:37.782831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:37.782904 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:37.783113 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:37.783119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:37.783172 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.783246 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.783398 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":40,"thread_start_us":89,"threads_started":1}
I20260925 14:58:37.785487 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:43668:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260925 14:58:37.785681 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:37.787053 15913 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69501516374c3bbfaf122a7fceab6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=362cf654149d4fd4a9a75edeca91c5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:37.787137 16053 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69501516374c3bbfaf122a7fceab6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=362cf654149d4fd4a9a75edeca91c5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:37.787153 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 5a69501516374c3bbfaf122a7fceab6b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=362cf654149d4fd4a9a75edeca91c5dc]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260925 14:58:37.787232 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69501516374c3bbfaf122a7fceab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.787264 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69501516374c3bbfaf122a7fceab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.787279 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a69501516374c3bbfaf122a7fceab6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.788677 16287 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: Bootstrap starting.
I20260925 14:58:37.788681 16291 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: Bootstrap starting.
I20260925 14:58:37.788853 16288 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: Bootstrap starting.
I20260925 14:58:37.789072 16287 tablet_bootstrap.cc:654] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.789072 16291 tablet_bootstrap.cc:654] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.789222 16288 tablet_bootstrap.cc:654] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.789839 16288 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: No bootstrap required, opened a new log
I20260925 14:58:37.789892 16288 ts_tablet_manager.cc:1397] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:37.789970 16287 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: No bootstrap required, opened a new log
I20260925 14:58:37.790024 16287 ts_tablet_manager.cc:1397] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:37.790043 16288 raft_consensus.cc:348] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790120 16288 raft_consensus.cc:374] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.790153 16288 raft_consensus.cc:729] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.790164 16287 raft_consensus.cc:348] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790200 16288 consensus_queue.cc:260] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790239 16287 raft_consensus.cc:374] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.790261 16287 raft_consensus.cc:729] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.790287 16288 ts_tablet_manager.cc:1428] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:37.790297 16287 consensus_queue.cc:260] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790369 16291 tablet_bootstrap.cc:492] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: No bootstrap required, opened a new log
I20260925 14:58:37.790380 16287 ts_tablet_manager.cc:1428] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:37.790410 16291 ts_tablet_manager.cc:1397] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.790414 16294 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.790480 16294 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.790470 16293 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.790529 16293 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.790548 16291 raft_consensus.cc:348] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790601 16291 raft_consensus.cc:374] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.790618 16291 raft_consensus.cc:729] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.790663 16291 consensus_queue.cc:260] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.790742 16291 ts_tablet_manager.cc:1428] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.790931 16295 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.791018 16295 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.885788 16294 raft_consensus.cc:482] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:37.885882 16294 raft_consensus.cc:504] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.886226 16294 leader_election.cc:288] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.888998 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.886443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:36439 (local address 127.8.13.195:40805)
0925 14:58:37.886489 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.886491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.886608 (+ 117us) server_negotiation.cc:407] Connection header received
0925 14:58:37.886653 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.886654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.886663 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.886709 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.886854 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.887303 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888340 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.888697 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888757 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.888766 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.888888 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.888925 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:58:37.889001 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.886451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56231 (local address 127.8.13.193:42717)
0925 14:58:37.886504 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.886506 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.886548 (+ 42us) server_negotiation.cc:407] Connection header received
0925 14:58:37.886619 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.886620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.886629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.886674 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:37.886813 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.887279 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888343 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.888699 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888760 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.888768 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.888895 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.888932 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 14:58:37.889027 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.886384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40805 (local address 127.8.13.194:36439)
0925 14:58:37.886515 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.886525 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.886598 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.886717 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.886717 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.886719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.886791 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.886793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.887371 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.887371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.888260 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.888262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.888761 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.888831 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.888836 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.888878 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.888925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":28,"thread_start_us":53,"threads_started":1}
I20260925 14:58:37.889035 16327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.886384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:42717 (local address 127.8.13.194:56231)
0925 14:58:37.886494 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.886504 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.886564 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.886686 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.886687 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.886688 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:37.886763 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.886765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.887349 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.887351 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.888249 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.888251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.888766 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.888766 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.888831 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.888834 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.888878 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.888925 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":75,"thread_start_us":54,"threads_started":1}
I20260925 14:58:37.889310 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69501516374c3bbfaf122a7fceab6b" candidate_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d" is_pre_election: true
I20260925 14:58:37.889312 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69501516374c3bbfaf122a7fceab6b" candidate_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204" is_pre_election: true
I20260925 14:58:37.889410 16213 raft_consensus.cc:2454] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4cb2a0b52347a7ae0f3101050a9059 in term 0.
I20260925 14:58:37.889422 15933 raft_consensus.cc:2454] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f4cb2a0b52347a7ae0f3101050a9059 in term 0.
I20260925 14:58:37.889575 16007 leader_election.cc:302] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, 46e410a3589e4edd91900c726c44d29d; no voters:
I20260925 14:58:37.889710 16294 raft_consensus.cc:2781] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:37.889745 16294 raft_consensus.cc:482] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:37.889765 16294 raft_consensus.cc:3037] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.890419 16294 raft_consensus.cc:504] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.890547 16294 leader_election.cc:288] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [CANDIDATE]: Term 1 election: Requested vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805)
I20260925 14:58:37.890698 16213 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69501516374c3bbfaf122a7fceab6b" candidate_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46e410a3589e4edd91900c726c44d29d"
I20260925 14:58:37.890691 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a69501516374c3bbfaf122a7fceab6b" candidate_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204"
I20260925 14:58:37.890765 15933 raft_consensus.cc:3037] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.890764 16213 raft_consensus.cc:3037] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:37.891302 16213 raft_consensus.cc:2454] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4cb2a0b52347a7ae0f3101050a9059 in term 1.
I20260925 14:58:37.891374 15933 raft_consensus.cc:2454] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f4cb2a0b52347a7ae0f3101050a9059 in term 1.
I20260925 14:58:37.891474 16007 leader_election.cc:302] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, 46e410a3589e4edd91900c726c44d29d; no voters:
I20260925 14:58:37.891577 16294 raft_consensus.cc:2781] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:37.891690 16294 raft_consensus.cc:686] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 LEADER]: Becoming Leader. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Running, Role: LEADER
I20260925 14:58:37.891750 16294 consensus_queue.cc:237] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.892081 16329 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: TxnStatusTablet state changed. Reason: New leader 2f4cb2a0b52347a7ae0f3101050a9059. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.892166 16329 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:37.892324 16331 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260925 14:58:37.892431 16331 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260925 14:58:37.892462 15776 catalog_manager.cc:5495] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:37.909550 16323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.906612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.0.0.1:46596)
0925 14:58:37.906708 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.906721 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.906777 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.906954 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.906955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.906956 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:37.907034 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.907036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.907967 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.907968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.908579 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.908581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.909034 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.909035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.909085 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.909101 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.909396 (+ 295us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.909404 (+ 8us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.909441 (+ 37us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.909489 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 14:58:37.909551 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.906693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46596 (local address 127.8.13.194:39955)
0925 14:58:37.906743 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.906745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.906758 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:58:37.906858 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.906859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.906878 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.906945 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:37.907099 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.907894 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.908650 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.908949 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.909012 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.909182 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.909335 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.909401 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.909445 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.909467 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 14:58:37.910722 16213 raft_consensus.cc:1261] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Refusing update from remote peer 2f4cb2a0b52347a7ae0f3101050a9059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.910745 15933 raft_consensus.cc:1261] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Refusing update from remote peer 2f4cb2a0b52347a7ae0f3101050a9059: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.910998 16294 consensus_queue.cc:1035] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.911123 16329 consensus_queue.cc:1035] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.911671 16295 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: TxnStatusTablet state changed. Reason: New leader 2f4cb2a0b52347a7ae0f3101050a9059. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.911769 16295 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.912125 16293 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: TxnStatusTablet state changed. Reason: New leader 2f4cb2a0b52347a7ae0f3101050a9059. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.912220 16293 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.912763 16294 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.912850 16294 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:37.913385 16295 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.913470 16295 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 46e410a3589e4edd91900c726c44d29d: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.913758 16294 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.913829 16294 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P 2f4cb2a0b52347a7ae0f3101050a9059: This TxnStatusTablet replica's current role is: LEADER
I20260925 14:58:37.914232 16293 tablet_replica.cc:439] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } } }
I20260925 14:58:37.914307 16293 tablet_replica.cc:442] T 5a69501516374c3bbfaf122a7fceab6b P d7ff56526a3c4594a92f4777fb7ed204: This TxnStatusTablet replica's current role is: FOLLOWER
I20260925 14:58:37.918722 16339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.916182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.8.13.193:45667)
0925 14:58:37.916315 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.916326 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.916381 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.916495 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.916496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.916497 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:37.916539 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.916540 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.917381 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.917381 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.917948 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.917949 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.918399 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.918400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.918439 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.918453 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.918591 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.918592 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.918619 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.918656 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260925 14:58:37.918715 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.916240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:45667 (local address 127.8.13.194:39955)
0925 14:58:37.916282 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.916284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.916379 (+ 95us) server_negotiation.cc:407] Connection header received
0925 14:58:37.916420 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.916420 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.916430 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.916492 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:37.916574 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.917306 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.918038 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.918340 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.918402 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.918513 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.918543 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.918583 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.918624 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.918642 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:37.920751 16073 raft_consensus.cc:1261] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Refusing update from remote peer d7ff56526a3c4594a92f4777fb7ed204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.920785 16213 raft_consensus.cc:1261] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Refusing update from remote peer d7ff56526a3c4594a92f4777fb7ed204: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:37.920970 16293 consensus_queue.cc:1035] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.921077 16306 consensus_queue.cc:1035] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:37.929013 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.926196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:35793 (local address 127.8.13.193:42717)
0925 14:58:37.926236 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.926238 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.926312 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:58:37.926366 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.926367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.926378 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.926439 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:37.926542 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.927292 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.928297 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.928601 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.928669 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.928789 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.928824 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.928868 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.928913 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.928935 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:58:37.929104 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.926125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:42717 (local address 127.8.13.194:35793)
0925 14:58:37.926258 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:37.926269 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:37.926326 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:37.926439 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:37.926440 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:37.926441 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:37.926498 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.926499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.927426 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.927427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.928223 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:37.928226 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.928665 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.928665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:37.928709 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.928723 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:37.928871 (+ 148us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:37.928872 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:37.928897 (+ 25us) client_negotiation.cc:240] Negotiation successful
0925 14:58:37.929022 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260925 14:58:37.937821 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:37.937918 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:37.938503 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Sending DeleteTablet for 3 replicas of tablet d45a818d2ef54814bccf6161e58f5489
I20260925 14:58:37.938781 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet d45a818d2ef54814bccf6161e58f5489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46580
I20260925 14:58:37.938776 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet d45a818d2ef54814bccf6161e58f5489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50088
I20260925 14:58:37.938871 16311 tablet_replica.cc:330] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: stopping tablet replica
I20260925 14:58:37.938889 16312 tablet_replica.cc:330] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: stopping tablet replica
I20260925 14:58:37.938921 16311 raft_consensus.cc:2229] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:37.938974 16311 raft_consensus.cc:2258] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.938972 16312 raft_consensus.cc:2229] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:37.939010 16312 raft_consensus.cc:2258] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.939173 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet d45a818d2ef54814bccf6161e58f5489 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:37 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33950
I20260925 14:58:37.939260 16313 tablet_replica.cc:330] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: stopping tablet replica
I20260925 14:58:37.939304 16313 raft_consensus.cc:2229] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:37.939332 16313 raft_consensus.cc:2258] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:37.939342 16312 ts_tablet_manager.cc:1905] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.939612 16311 ts_tablet_manager.cc:1905] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.941118 16311 ts_tablet_manager.cc:1918] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:37.941179 16311 log.cc:1196] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/d45a818d2ef54814bccf6161e58f5489
I20260925 14:58:37.941227 16312 ts_tablet_manager.cc:1918] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:37.941286 16312 log.cc:1196] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/d45a818d2ef54814bccf6161e58f5489
I20260925 14:58:37.941385 16311 ts_tablet_manager.cc:1939] T d45a818d2ef54814bccf6161e58f5489 P d7ff56526a3c4594a92f4777fb7ed204: Deleting consensus metadata
I20260925 14:58:37.941478 16312 ts_tablet_manager.cc:1939] T d45a818d2ef54814bccf6161e58f5489 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting consensus metadata
I20260925 14:58:37.941762 16313 ts_tablet_manager.cc:1905] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:37.941800 15762 catalog_manager.cc:4841] TS d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717): tablet d45a818d2ef54814bccf6161e58f5489 (table test-table [id=2bb0efb0c154494ca51911133243a684]) successfully deleted
I20260925 14:58:37.941921 15763 catalog_manager.cc:4841] TS 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955): tablet d45a818d2ef54814bccf6161e58f5489 (table test-table [id=2bb0efb0c154494ca51911133243a684]) successfully deleted
I20260925 14:58:37.942975 16313 ts_tablet_manager.cc:1918] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:37.943059 16313 log.cc:1196] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/d45a818d2ef54814bccf6161e58f5489
I20260925 14:58:37.943264 16313 ts_tablet_manager.cc:1939] T d45a818d2ef54814bccf6161e58f5489 P 46e410a3589e4edd91900c726c44d29d: Deleting consensus metadata
I20260925 14:58:37.943588 15763 catalog_manager.cc:4841] TS 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805): tablet d45a818d2ef54814bccf6161e58f5489 (table test-table [id=2bb0efb0c154494ca51911133243a684]) successfully deleted
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:58:37.962579 15833 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:37.956775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43670 (local address 127.8.13.254:33829)
0925 14:58:37.956826 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.956828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.956912 (+ 84us) server_negotiation.cc:407] Connection header received
0925 14:58:37.956992 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.956993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.957002 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.957050 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.957786 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.958273 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.959287 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.959464 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.959508 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.962511 (+ 3003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43670: BlockingRecv error: recv got EOF from 127.0.0.1:43670 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 25 14:58:37 dist-test-slave-15hz krb5kdc[15731](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM
I20260925 14:58:37.969575 15833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:37.963401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43684 (local address 127.8.13.254:33829)
0925 14:58:37.963466 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:37.963468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:37.963476 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:37.963500 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:37.963501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:37.963510 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:37.963587 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:37.964008 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.964485 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.965142 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:37.965323 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:37.965373 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:37.967333 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:37.967334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:37.967335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:37.967342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:37.968624 (+ 1282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.968909 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.968910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.968910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.968958 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:37.969129 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:37.969130 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:37.969130 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:37.969230 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:37.969311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:37.969413 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 14:58:37.969484 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:37.971083 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:37.971248 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:37.972544 16053 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7d40f7d23b4492af77976693a3d475 (DEFAULT_TABLE table=test-table [id=d9f49cbb46074e8badfb020a7a1773c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.972613 16193 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7d40f7d23b4492af77976693a3d475 (DEFAULT_TABLE table=test-table [id=d9f49cbb46074e8badfb020a7a1773c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.972692 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7d40f7d23b4492af77976693a3d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.972766 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7d40f7d23b4492af77976693a3d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.973618 16288 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Bootstrap starting.
I20260925 14:58:37.973672 15913 tablet_service.cc:1467] Processing CreateTablet for tablet 0e7d40f7d23b4492af77976693a3d475 (DEFAULT_TABLE table=test-table [id=d9f49cbb46074e8badfb020a7a1773c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:37.973779 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e7d40f7d23b4492af77976693a3d475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:37.974016 16288 tablet_bootstrap.cc:654] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.974239 16291 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Bootstrap starting.
I20260925 14:58:37.974596 16291 tablet_bootstrap.cc:654] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.975101 16287 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Bootstrap starting.
I20260925 14:58:37.975358 16291 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: No bootstrap required, opened a new log
I20260925 14:58:37.975394 16287 tablet_bootstrap.cc:654] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:37.975409 16291 ts_tablet_manager.cc:1397] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:37.975574 16291 raft_consensus.cc:348] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.975659 16291 raft_consensus.cc:374] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.975695 16291 raft_consensus.cc:729] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.975736 16291 consensus_queue.cc:260] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.975828 16291 ts_tablet_manager.cc:1428] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:37.976015 16288 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: No bootstrap required, opened a new log
I20260925 14:58:37.976063 16288 ts_tablet_manager.cc:1397] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:37.976100 16287 tablet_bootstrap.cc:492] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: No bootstrap required, opened a new log
I20260925 14:58:37.976141 16287 ts_tablet_manager.cc:1397] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:37.976202 16288 raft_consensus.cc:348] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.976235 16287 raft_consensus.cc:348] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.976280 16288 raft_consensus.cc:374] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.976289 16287 raft_consensus.cc:374] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:37.976300 16288 raft_consensus.cc:729] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.976307 16287 raft_consensus.cc:729] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Initialized, Role: FOLLOWER
I20260925 14:58:37.976348 16288 consensus_queue.cc:260] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.976351 16287 consensus_queue.cc:260] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:37.976433 16288 ts_tablet_manager.cc:1428] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:37.976435 16287 ts_tablet_manager.cc:1428] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:38.059345 16295 raft_consensus.cc:482] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:38.059450 16295 raft_consensus.cc:504] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.059818 16295 leader_election.cc:288] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955)
I20260925 14:58:38.062726 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.060053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:53745 (local address 127.8.13.194:39955)
0925 14:58:38.060121 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:38.060123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:38.060254 (+ 131us) server_negotiation.cc:407] Connection header received
0925 14:58:38.060305 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:38.060305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:38.060315 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:38.060366 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:38.060506 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.060959 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.061951 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.062411 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.062466 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.062473 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:38.062619 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 14:58:38.062651 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 14:58:38.062767 16361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.059991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.8.13.195:53745)
0925 14:58:38.060180 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:38.060190 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:38.060253 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:38.060368 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:38.060369 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:38.060371 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:38.060442 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.060444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.061054 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.061055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.061869 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.061871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.062478 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.062479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.062553 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.062558 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:58:38.062615 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 14:58:38.062669 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":3,"thread_start_us":63,"threads_started":1}
I20260925 14:58:38.062985 16360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.059991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:42717 (local address 127.8.13.195:38259)
0925 14:58:38.060137 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:38.060149 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:38.060209 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:38.060326 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:38.060327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:38.060329 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:38.060405 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.060408 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.061049 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.061050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.062085 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.062087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.062692 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.062693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.062780 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.062784 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:38.062871 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 14:58:38.062910 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":64,"threads_started":1}
I20260925 14:58:38.062985 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.060113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:38259 (local address 127.8.13.193:42717)
0925 14:58:38.060201 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:38.060204 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:38.060215 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:38.060252 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:38.060253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:38.060266 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:38.060317 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:38.060470 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.060960 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.062187 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.062589 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.062650 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.062657 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:38.062882 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 14:58:38.062927 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:58:38.063002 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7d40f7d23b4492af77976693a3d475" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" is_pre_election: true
I20260925 14:58:38.063093 16073 raft_consensus.cc:2454] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 0.
I20260925 14:58:38.063298 16147 leader_election.cc:302] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, 46e410a3589e4edd91900c726c44d29d; no voters:
I20260925 14:58:38.063328 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7d40f7d23b4492af77976693a3d475" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204" is_pre_election: true
I20260925 14:58:38.063428 15933 raft_consensus.cc:2454] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 0.
I20260925 14:58:38.063431 16295 raft_consensus.cc:2781] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:38.063465 16295 raft_consensus.cc:482] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:38.063483 16295 raft_consensus.cc:3037] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.064175 16295 raft_consensus.cc:504] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.064286 16295 leader_election.cc:288] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 election: Requested vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955)
I20260925 14:58:38.064433 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7d40f7d23b4492af77976693a3d475" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204"
I20260925 14:58:38.064437 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e7d40f7d23b4492af77976693a3d475" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
I20260925 14:58:38.064508 15933 raft_consensus.cc:3037] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.064512 16073 raft_consensus.cc:3037] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.065162 15933 raft_consensus.cc:2454] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 1.
I20260925 14:58:38.065236 16073 raft_consensus.cc:2454] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 1.
I20260925 14:58:38.065320 16146 leader_election.cc:302] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e410a3589e4edd91900c726c44d29d, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:38.065397 16295 raft_consensus.cc:2781] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:38.065506 16295 raft_consensus.cc:686] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 1 LEADER]: Becoming Leader. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Running, Role: LEADER
I20260925 14:58:38.065567 16295 consensus_queue.cc:237] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.066238 15776 catalog_manager.cc:5495] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e410a3589e4edd91900c726c44d29d (127.8.13.195). New cstate: current_term: 1 leader_uuid: "46e410a3589e4edd91900c726c44d29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:38.092337 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.090034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.8.13.195:40805)
0925 14:58:38.090086 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:38.090088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:38.090096 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:38.090138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:38.090139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:38.090156 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:38.090215 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:38.090328 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.090836 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.091650 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.091837 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.091891 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.092041 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:38.092139 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:38.092197 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:38.092241 (+ 44us) server_negotiation.cc:299] Negotiation successful
0925 14:58:38.092259 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 14:58:38.093271 15933 raft_consensus.cc:1261] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Refusing update from remote peer 46e410a3589e4edd91900c726c44d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:38.093269 16073 raft_consensus.cc:1261] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Refusing update from remote peer 46e410a3589e4edd91900c726c44d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:38.093550 16295 consensus_queue.cc:1035] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:38.093703 16362 consensus_queue.cc:1035] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:38.096752 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:38.096843 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:38.097406 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Sending DeleteTablet for 3 replicas of tablet 0e7d40f7d23b4492af77976693a3d475
I20260925 14:58:38.097698 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7d40f7d23b4492af77976693a3d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46580
I20260925 14:58:38.097702 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7d40f7d23b4492af77976693a3d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50088
I20260925 14:58:38.097751 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e7d40f7d23b4492af77976693a3d475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33950
I20260925 14:58:38.097810 16311 tablet_replica.cc:330] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: stopping tablet replica
I20260925 14:58:38.097856 16311 raft_consensus.cc:2229] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:38.097848 16312 tablet_replica.cc:330] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: stopping tablet replica
I20260925 14:58:38.097895 16311 raft_consensus.cc:2258] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.097898 16312 raft_consensus.cc:2229] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:38.097947 16312 raft_consensus.cc:2258] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.098017 16313 tablet_replica.cc:330] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: stopping tablet replica
I20260925 14:58:38.098068 16313 raft_consensus.cc:2229] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:38.098140 16313 raft_consensus.cc:2258] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.098197 16311 ts_tablet_manager.cc:1905] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.098580 16313 ts_tablet_manager.cc:1905] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.099416 15776 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:38.099578 15776 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:38.099818 16311 ts_tablet_manager.cc:1918] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:38.099870 16311 log.cc:1196] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/0e7d40f7d23b4492af77976693a3d475
I20260925 14:58:38.100028 16313 ts_tablet_manager.cc:1918] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:38.100061 16311 ts_tablet_manager.cc:1939] T 0e7d40f7d23b4492af77976693a3d475 P d7ff56526a3c4594a92f4777fb7ed204: Deleting consensus metadata
I20260925 14:58:38.100076 16313 log.cc:1196] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/0e7d40f7d23b4492af77976693a3d475
I20260925 14:58:38.100275 16313 ts_tablet_manager.cc:1939] T 0e7d40f7d23b4492af77976693a3d475 P 46e410a3589e4edd91900c726c44d29d: Deleting consensus metadata
I20260925 14:58:38.100373 15762 catalog_manager.cc:4841] TS d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717): tablet 0e7d40f7d23b4492af77976693a3d475 (table test-table [id=d9f49cbb46074e8badfb020a7a1773c2]) successfully deleted
I20260925 14:58:38.100589 16312 ts_tablet_manager.cc:1905] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.100677 15913 tablet_service.cc:1467] Processing CreateTablet for tablet a4e676067ef14f9bb95e4b382337bc72 (DEFAULT_TABLE table=test-table [id=53803d09c1504600b4e7c956b42878c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:38.100772 15913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e676067ef14f9bb95e4b382337bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:38.101540 16287 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: Bootstrap starting.
I20260925 14:58:38.101778 15763 catalog_manager.cc:4841] TS 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805): tablet 0e7d40f7d23b4492af77976693a3d475 (table test-table [id=d9f49cbb46074e8badfb020a7a1773c2]) successfully deleted
I20260925 14:58:38.101835 16287 tablet_bootstrap.cc:654] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:38.101969 16312 ts_tablet_manager.cc:1918] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:38.102015 16312 log.cc:1196] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/0e7d40f7d23b4492af77976693a3d475
I20260925 14:58:38.102109 16193 tablet_service.cc:1467] Processing CreateTablet for tablet a4e676067ef14f9bb95e4b382337bc72 (DEFAULT_TABLE table=test-table [id=53803d09c1504600b4e7c956b42878c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:38.102197 16312 ts_tablet_manager.cc:1939] T 0e7d40f7d23b4492af77976693a3d475 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting consensus metadata
I20260925 14:58:38.102201 16193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e676067ef14f9bb95e4b382337bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:38.102634 15763 catalog_manager.cc:4841] TS 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955): tablet 0e7d40f7d23b4492af77976693a3d475 (table test-table [id=d9f49cbb46074e8badfb020a7a1773c2]) successfully deleted
I20260925 14:58:38.103096 16291 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Bootstrap starting.
I20260925 14:58:38.103124 16053 tablet_service.cc:1467] Processing CreateTablet for tablet a4e676067ef14f9bb95e4b382337bc72 (DEFAULT_TABLE table=test-table [id=53803d09c1504600b4e7c956b42878c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:38.103216 16053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4e676067ef14f9bb95e4b382337bc72. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:38.103456 16291 tablet_bootstrap.cc:654] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:38.104131 16291 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: No bootstrap required, opened a new log
I20260925 14:58:38.104172 16291 ts_tablet_manager.cc:1397] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:38.104209 16288 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Bootstrap starting.
I20260925 14:58:38.104326 16291 raft_consensus.cc:348] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.104404 16291 raft_consensus.cc:374] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:38.104434 16291 raft_consensus.cc:729] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Initialized, Role: FOLLOWER
I20260925 14:58:38.104473 16291 consensus_queue.cc:260] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.104501 16288 tablet_bootstrap.cc:654] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:38.104561 16291 ts_tablet_manager.cc:1428] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:38.104955 16287 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: No bootstrap required, opened a new log
I20260925 14:58:38.105006 16287 ts_tablet_manager.cc:1397] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260925 14:58:38.105120 16288 tablet_bootstrap.cc:492] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: No bootstrap required, opened a new log
I20260925 14:58:38.105105 16287 raft_consensus.cc:348] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.105167 16287 raft_consensus.cc:374] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:38.105165 16288 ts_tablet_manager.cc:1397] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260925 14:58:38.105194 16287 raft_consensus.cc:729] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7ff56526a3c4594a92f4777fb7ed204, State: Initialized, Role: FOLLOWER
I20260925 14:58:38.105230 16287 consensus_queue.cc:260] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.105278 16288 raft_consensus.cc:348] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.105315 16287 ts_tablet_manager.cc:1428] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:38.105342 16288 raft_consensus.cc:374] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:38.105365 16288 raft_consensus.cc:729] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4cb2a0b52347a7ae0f3101050a9059, State: Initialized, Role: FOLLOWER
I20260925 14:58:38.105408 16288 consensus_queue.cc:260] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.105509 16288 ts_tablet_manager.cc:1428] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:38.164357 16362 raft_consensus.cc:482] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:38.164467 16362 raft_consensus.cc:504] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.164664 16362 leader_election.cc:288] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955)
I20260925 14:58:38.164865 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e676067ef14f9bb95e4b382337bc72" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" is_pre_election: true
I20260925 14:58:38.164973 16073 raft_consensus.cc:2454] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 0.
I20260925 14:58:38.164949 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e676067ef14f9bb95e4b382337bc72" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204" is_pre_election: true
I20260925 14:58:38.165037 15933 raft_consensus.cc:2454] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 0.
I20260925 14:58:38.165166 16146 leader_election.cc:302] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46e410a3589e4edd91900c726c44d29d, d7ff56526a3c4594a92f4777fb7ed204; no voters:
I20260925 14:58:38.165259 16362 raft_consensus.cc:2781] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:38.165306 16362 raft_consensus.cc:482] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:38.165331 16362 raft_consensus.cc:3037] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.166003 16362 raft_consensus.cc:504] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.166113 16362 leader_election.cc:288] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 election: Requested vote from peers d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717), 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955)
I20260925 14:58:38.166405 16073 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e676067ef14f9bb95e4b382337bc72" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f4cb2a0b52347a7ae0f3101050a9059"
I20260925 14:58:38.166440 15933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4e676067ef14f9bb95e4b382337bc72" candidate_uuid: "46e410a3589e4edd91900c726c44d29d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7ff56526a3c4594a92f4777fb7ed204"
I20260925 14:58:38.166487 16073 raft_consensus.cc:3037] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.166502 15933 raft_consensus.cc:3037] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:38.167219 15933 raft_consensus.cc:2454] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 1.
I20260925 14:58:38.167219 16073 raft_consensus.cc:2454] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46e410a3589e4edd91900c726c44d29d in term 1.
I20260925 14:58:38.167372 16147 leader_election.cc:302] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f4cb2a0b52347a7ae0f3101050a9059, 46e410a3589e4edd91900c726c44d29d; no voters:
I20260925 14:58:38.167451 16362 raft_consensus.cc:2781] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:38.167488 16362 raft_consensus.cc:686] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 1 LEADER]: Becoming Leader. State: Replica: 46e410a3589e4edd91900c726c44d29d, State: Running, Role: LEADER
I20260925 14:58:38.167546 16362 consensus_queue.cc:237] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } }
I20260925 14:58:38.168138 15774 catalog_manager.cc:5495] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d reported cstate change: term changed from 0 to 1, leader changed from <none> to 46e410a3589e4edd91900c726c44d29d (127.8.13.195). New cstate: current_term: 1 leader_uuid: "46e410a3589e4edd91900c726c44d29d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46e410a3589e4edd91900c726c44d29d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40805 } health_report { overall_health: HEALTHY } } }
I20260925 14:58:38.183137 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.180590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:60689 (local address 127.8.13.194:39955)
0925 14:58:38.180633 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:38.180635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:38.180746 (+ 111us) server_negotiation.cc:407] Connection header received
0925 14:58:38.180816 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:38.180817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:38.180828 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:38.180870 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:38.181007 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.181465 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.182504 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.182695 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.182742 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.182931 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:38.182957 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:38.182987 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:38.183056 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:58:38.183073 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 14:58:38.183199 16375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.180518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:39955 (local address 127.8.13.195:60689)
0925 14:58:38.180687 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:38.180699 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:38.180762 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:38.180892 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:38.180893 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:38.180895 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:38.180952 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.180954 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.181529 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.181531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.182416 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.182420 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.182764 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.182765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.182841 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.182858 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:38.183004 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:38.183006 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:38.183053 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:38.183100 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":91,"threads_started":1}
I20260925 14:58:38.185145 16073 raft_consensus.cc:1261] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Refusing update from remote peer 46e410a3589e4edd91900c726c44d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:38.185138 15933 raft_consensus.cc:1261] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Refusing update from remote peer 46e410a3589e4edd91900c726c44d29d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:38.185392 16367 consensus_queue.cc:1035] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7ff56526a3c4594a92f4777fb7ed204" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 42717 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:38.185526 16367 consensus_queue.cc:1035] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f4cb2a0b52347a7ae0f3101050a9059" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 39955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:38.192632 16286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.190275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:37683 (local address 127.8.13.195:40805)
0925 14:58:38.190313 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:38.190314 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:38.190322 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:38.190339 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:38.190340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:38.190350 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:38.190399 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:38.190506 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.190967 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.192012 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.192233 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.192283 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.192442 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:38.192467 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:38.192497 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:38.192548 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 14:58:38.192563 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:38.192678 16343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.190124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40805 (local address 127.8.13.194:37683)
0925 14:58:38.190168 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:38.190182 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:38.190219 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:38.190407 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:38.190408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:38.190410 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0925 14:58:38.190454 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.190455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.191050 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.191051 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.191905 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:38.191909 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:38.192306 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:38.192308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:38.192368 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:38.192376 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0925 14:58:38.192501 (+ 125us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0925 14:58:38.192502 (+ 1us) client_negotiation.cc:769] Sending connection context
0925 14:58:38.192545 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 14:58:38.192596 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 14:58:38.198354 15776 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:38.198472 15776 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43684:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:38.199105 15776 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5248a29df19e4193b0567b1bb5477836: Sending DeleteTablet for 3 replicas of tablet a4e676067ef14f9bb95e4b382337bc72
I20260925 14:58:38.199328 16053 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e676067ef14f9bb95e4b382337bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46580
I20260925 14:58:38.199354 15913 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e676067ef14f9bb95e4b382337bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:50088
I20260925 14:58:38.199374 16193 tablet_service.cc:1514] Processing DeleteTablet for tablet a4e676067ef14f9bb95e4b382337bc72 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:38 UTC) from {username='oryx', principal='oryx/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33950
I20260925 14:58:38.199432 16312 tablet_replica.cc:330] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: stopping tablet replica
I20260925 14:58:38.199469 16311 tablet_replica.cc:330] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: stopping tablet replica
I20260925 14:58:38.199494 16312 raft_consensus.cc:2229] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:38.199548 16312 raft_consensus.cc:2258] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.199548 16311 raft_consensus.cc:2229] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:38.199579 16311 raft_consensus.cc:2258] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.199609 16313 tablet_replica.cc:330] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: stopping tablet replica
I20260925 14:58:38.199679 16313 raft_consensus.cc:2229] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:38.199738 16313 raft_consensus.cc:2258] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:38.200009 16312 ts_tablet_manager.cc:1905] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.200026 16313 ts_tablet_manager.cc:1905] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.201503 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15848
I20260925 14:58:38.201607 16312 ts_tablet_manager.cc:1918] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:38.201653 16312 log.cc:1196] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/a4e676067ef14f9bb95e4b382337bc72
I20260925 14:58:38.201671 16311 ts_tablet_manager.cc:1905] T a4e676067ef14f9bb95e4b382337bc72 P d7ff56526a3c4594a92f4777fb7ed204: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:38.201857 16312 ts_tablet_manager.cc:1939] T a4e676067ef14f9bb95e4b382337bc72 P 2f4cb2a0b52347a7ae0f3101050a9059: Deleting consensus metadata
I20260925 14:58:38.202499 16313 ts_tablet_manager.cc:1918] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260925 14:58:38.202549 16313 log.cc:1196] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/a4e676067ef14f9bb95e4b382337bc72
I20260925 14:58:38.202720 16313 ts_tablet_manager.cc:1939] T a4e676067ef14f9bb95e4b382337bc72 P 46e410a3589e4edd91900c726c44d29d: Deleting consensus metadata
I20260925 14:58:38.203042 15763 catalog_manager.cc:4841] TS 46e410a3589e4edd91900c726c44d29d (127.8.13.195:40805): tablet a4e676067ef14f9bb95e4b382337bc72 (table test-table [id=53803d09c1504600b4e7c956b42878c4]) successfully deleted
I20260925 14:58:38.203626 15763 catalog_manager.cc:4841] TS 2f4cb2a0b52347a7ae0f3101050a9059 (127.8.13.194:39955): tablet a4e676067ef14f9bb95e4b382337bc72 (table test-table [id=53803d09c1504600b4e7c956b42878c4]) successfully deleted
W20260925 14:58:38.208521 15762 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:42717 (error 108)
I20260925 14:58:38.209002 16379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:38.208871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:42717 (local address 127.0.0.1:50114)
0925 14:58:38.208936 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:38.208966 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:42717: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52,"thread_start_us":36,"threads_started":1}
W20260925 14:58:38.209091 15762 catalog_manager.cc:4584] TS d7ff56526a3c4594a92f4777fb7ed204 (127.8.13.193:42717): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4e676067ef14f9bb95e4b382337bc72: Network error: Client connection negotiation failed: client connection to 127.8.13.193:42717: connect: Connection refused (error 111)
I20260925 14:58:38.209388 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15988
I20260925 14:58:38.216991 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16128
I20260925 14:58:38.221689 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 15744
2026-09-25T14:58:38Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2542 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:38 dist-test-slave-15hz krb5kdc[16382](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:38 dist-test-slave-15hz krb5kdc[16382](info): set up 2 sockets
Sep 25 14:58:38 dist-test-slave-15hz krb5kdc[16382](info): commencing operation
krb5kdc: starting...
W20260925 14:58:39.175740 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:39 dist-test-slave-15hz krb5kdc[16382](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:39.287335 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34379
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:34379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:39.367162 16395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:39.367365 16395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:39.367386 16395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:39.368808 16395 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:39.368858 16395 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:39.368871 16395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:39.368882 16395 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:39.368903 16395 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:39.370611 16395 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:34379
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16395
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:39.370859 16395 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:39.371078 16395 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:39.375723 16400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.375741 16401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:39.375741 16403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:39.375875 16395 server_base.cc:1034] running on GCE node
I20260925 14:58:39.376166 16395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:39.376442 16395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:39.377614 16395 hybrid_clock.cc:648] HybridClock initialized: now 1790348319377581 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-25T14:58:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1163 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:39 dist-test-slave-15hz krb5kdc[16408](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:39 dist-test-slave-15hz krb5kdc[16408](info): set up 2 sockets
Sep 25 14:58:39 dist-test-slave-15hz krb5kdc[16408](info): commencing operation
krb5kdc: starting...
W20260925 14:58:40.335891 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:40.456243 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44567
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:40.535871 16421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:40.536093 16421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:40.536116 16421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:40.537228 16421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:40.537261 16421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:40.537274 16421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:40.537285 16421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:40.537297 16421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:40.538700 16421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:40.538894 16421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:40.539072 16421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:40.543789 16427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.543779 16426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.543877 16421 server_base.cc:1034] running on GCE node
W20260925 14:58:40.543784 16429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.544242 16421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:40.544497 16421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:40.545641 16421 hybrid_clock.cc:648] HybridClock initialized: now 1790348320545622 us; error 38 us; skew 500 ppm
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:40.548086 16421 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:40.548519 16421 webserver.cc:460] Webserver started at http://127.8.13.254:32835/ using document root <none> and password file <none>
I20260925 14:58:40.548722 16421 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:40.548767 16421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:40.548890 16421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:40.549947 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "16bd47926e024c138c9be2bd9d346fdd"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "779defbd4d6fe070540e5f60042580e4"
server_key_iv: "372f9ab603f603c017ce72338329d820"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.550274 16421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "16bd47926e024c138c9be2bd9d346fdd"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "779defbd4d6fe070540e5f60042580e4"
server_key_iv: "372f9ab603f603c017ce72338329d820"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.551676 16421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:40.552465 16436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.552645 16421 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:40.552729 16421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "16bd47926e024c138c9be2bd9d346fdd"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "779defbd4d6fe070540e5f60042580e4"
server_key_iv: "372f9ab603f603c017ce72338329d820"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.552788 16421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:40.575727 16421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:40.577893 16421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:40.578011 16421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:40.581779 16421 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44567
I20260925 14:58:40.581815 16488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44567 every 8 connection(s)
I20260925 14:58:40.582170 16421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:40.583048 16489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:40.585403 16489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Bootstrap starting.
I20260925 14:58:40.585991 16489 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:40.586253 16489 log.cc:824] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:40.586990 16489 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: No bootstrap required, opened a new log
I20260925 14:58:40.588868 16489 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:40.589013 16489 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:40.589057 16489 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd47926e024c138c9be2bd9d346fdd, State: Initialized, Role: FOLLOWER
I20260925 14:58:40.589206 16489 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:40.589267 16489 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:40.589288 16489 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:40.589334 16489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:40.589849 16489 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:40.589941 16489 leader_election.cc:302] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16bd47926e024c138c9be2bd9d346fdd; no voters:
I20260925 14:58:40.590109 16489 leader_election.cc:288] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:40.590148 16492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:40.590276 16492 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 LEADER]: Becoming Leader. State: Replica: 16bd47926e024c138c9be2bd9d346fdd, State: Running, Role: LEADER
I20260925 14:58:40.590370 16489 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:40.590360 16492 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:40.590768 16494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16bd47926e024c138c9be2bd9d346fdd. Latest consensus state: current_term: 1 leader_uuid: "16bd47926e024c138c9be2bd9d346fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } } }
I20260925 14:58:40.591061 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16421
I20260925 14:58:40.591158 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:40.591346 8247 external_mini_cluster.cc:1428] Setting key 5db7c5976745ca5a7e24754a2e0faace
I20260925 14:58:40.590903 16493 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16bd47926e024c138c9be2bd9d346fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } } }
I20260925 14:58:40.591596 16493 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:40.591835 16494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: This master's current role is: LEADER
W20260925 14:58:40.592116 16507 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:40.592195 16507 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:40.592271 16508 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:40.592440 16508 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:40.593804 16508 catalog_manager.cc:1347] Generated new cluster ID: 80ba7e2b5c39431b845e1ef03dcb654d
I20260925 14:58:40.593853 16508 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:40.603012 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.592037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48334 (local address 127.8.13.254:44567)
0925 14:58:40.592322 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.592333 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.592364 (+ 31us) server_negotiation.cc:407] Connection header received
0925 14:58:40.592490 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.592498 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.592557 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.592694 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.593503 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.594888 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.595993 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.596595 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.596690 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.599403 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.599408 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.599408 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.599423 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.601360 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.601744 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.601748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.601748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.601867 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.602047 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.602048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.602049 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.602372 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.602543 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.602612 (+ 69us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.602765 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":84,"threads_started":1}
I20260925 14:58:40.605458 16508 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:40.605964 16508 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:40.612520 16508 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Generated new TSK 0
I20260925 14:58:40.612713 16508 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:40.646771 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:40.726935 16515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:40.727159 16515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:40.727180 16515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:40.728370 16515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:40.728410 16515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:40.728437 16515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:40.729853 16515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16515
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:40.730088 16515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:40.730298 16515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:40.735029 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.735054 16520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.735167 16523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.735303 16515 server_base.cc:1034] running on GCE node
I20260925 14:58:40.735488 16515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:40.735742 16515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:40.736886 16515 hybrid_clock.cc:648] HybridClock initialized: now 1790348320736871 us; error 29 us; skew 500 ppm
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:40.742343 16515 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:40.742808 16515 webserver.cc:460] Webserver started at http://127.8.13.193:41887/ using document root <none> and password file <none>
I20260925 14:58:40.743002 16515 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:40.743043 16515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:40.743139 16515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:40.744226 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "454ba67e8fee499c9bd7964826408b40"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "ef0b156ef12a67464fdd6161617ef2cb"
server_key_iv: "5a5485d27036dbbd663e3991efffc603"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.744525 16515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "454ba67e8fee499c9bd7964826408b40"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "ef0b156ef12a67464fdd6161617ef2cb"
server_key_iv: "5a5485d27036dbbd663e3991efffc603"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.745752 16515 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:40.746521 16530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.746698 16515 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:40.746780 16515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "454ba67e8fee499c9bd7964826408b40"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "ef0b156ef12a67464fdd6161617ef2cb"
server_key_iv: "5a5485d27036dbbd663e3991efffc603"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.746850 16515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:40.757815 16515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:40.759868 16515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:40.759995 16515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:40.760221 16515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:40.760525 16515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:40.760560 16515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.760586 16515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:40.760609 16515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.766039 16515 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:38613
I20260925 14:58:40.766124 16643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:38613 every 8 connection(s)
I20260925 14:58:40.766513 16515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:40.771864 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16515
I20260925 14:58:40.771961 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:40.772146 8247 external_mini_cluster.cc:1428] Setting key c5213f44db004d6c65f74b4b4b54d8e1
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:40.776486 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.767286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:46207 (local address 127.8.13.254:44567)
0925 14:58:40.767367 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.767370 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.767587 (+ 217us) server_negotiation.cc:407] Connection header received
0925 14:58:40.768111 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.768112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.768127 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.768194 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.769185 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.769718 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.770774 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.770978 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.771029 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.773508 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.773509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.773509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.773517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.775017 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.775482 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.775483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.775483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.775586 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.775842 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.775843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.775843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.775950 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.775995 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.776319 (+ 324us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.776396 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 14:58:40.776749 16646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.767209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:46207)
0925 14:58:40.767435 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:40.767478 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:40.767983 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:40.768259 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:40.768263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:40.768831 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:40.769081 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.769085 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.769812 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.769813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.770697 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.770700 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.771095 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.771096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.771216 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.771834 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:40.771842 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:40.773416 (+ 1574us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:40.775091 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.775099 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.775127 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.775402 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.775649 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.775650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.775650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.775785 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.775998 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:40.776004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:40.776173 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 14:58:40.776302 (+ 129us) client_negotiation.cc:240] Negotiation successful
0925 14:58:40.776487 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":77,"threads_started":1}
I20260925 14:58:40.777297 16644 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44567
I20260925 14:58:40.777393 16644 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:40.777571 16644 heartbeater.cc:511] Master 127.8.13.254:44567 requested a full tablet report, sending...
I20260925 14:58:40.778155 16453 ts_manager.cc:194] Registered new tserver with Master: 454ba67e8fee499c9bd7964826408b40 (127.8.13.193:38613)
I20260925 14:58:40.778782 16453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:46207
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:40.799049 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:40.878943 16649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:40.879179 16649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:40.879210 16649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:40.880442 16649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:40.880486 16649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:40.880519 16649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:40.881948 16649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:40.882206 16649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:40.882417 16649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:40.887212 16654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.887281 16655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:40.887220 16657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:40.887416 16649 server_base.cc:1034] running on GCE node
I20260925 14:58:40.887699 16649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:40.887936 16649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:40.889114 16649 hybrid_clock.cc:648] HybridClock initialized: now 1790348320889092 us; error 38 us; skew 500 ppm
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:40.891517 16649 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:40.892022 16649 webserver.cc:460] Webserver started at http://127.8.13.194:36433/ using document root <none> and password file <none>
I20260925 14:58:40.892215 16649 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:40.892253 16649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:40.892343 16649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:40.893356 16649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "312022ce3a7f4652b8eedf480cb75a7b"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "423037ca838a76de444f5295c93633fb"
server_key_iv: "bd3760c9384fbb18711bdc62623cb82e"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.893646 16649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "312022ce3a7f4652b8eedf480cb75a7b"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "423037ca838a76de444f5295c93633fb"
server_key_iv: "bd3760c9384fbb18711bdc62623cb82e"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.894887 16649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 14:58:40.895716 16664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.895884 16649 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:40.895977 16649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "312022ce3a7f4652b8eedf480cb75a7b"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "423037ca838a76de444f5295c93633fb"
server_key_iv: "bd3760c9384fbb18711bdc62623cb82e"
server_key_version: "encryptionkey@0"
I20260925 14:58:40.896036 16649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:40.918857 16649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:40.921023 16649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:40.921137 16649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:40.921325 16649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:40.921622 16649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:40.921654 16649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.921679 16649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:40.921701 16649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:40.927211 16649 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45879
I20260925 14:58:40.927254 16777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45879 every 8 connection(s)
I20260925 14:58:40.927589 16649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:40 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:40.933867 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16649
I20260925 14:58:40.933990 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:40.934180 8247 external_mini_cluster.cc:1428] Setting key 681a1de0a9a05cf46e6578bfe31c19d1
I20260925 14:58:40.937172 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.928363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:48053 (local address 127.8.13.254:44567)
0925 14:58:40.928422 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:40.928424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:40.928590 (+ 166us) server_negotiation.cc:407] Connection header received
0925 14:58:40.929107 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:40.929108 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:40.929135 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:40.929209 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:40.930183 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.930694 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.931769 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.931965 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.932022 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.934340 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:40.934342 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:40.934342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:40.934350 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:40.935819 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.936205 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.936206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.936207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.936271 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:40.936542 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:40.936543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:40.936543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:40.936654 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:40.936705 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:40.937016 (+ 311us) server_negotiation.cc:299] Negotiation successful
0925 14:58:40.937091 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 14:58:40.937415 16780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:40.928287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:48053)
0925 14:58:40.928455 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:40.928501 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:40.928983 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:40.929274 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:40.929277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:40.929862 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:40.930066 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.930071 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.930804 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.930805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.931682 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:40.931685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:40.932063 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:40.932064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:40.932181 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:40.932723 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:40.932732 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:40.934250 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:40.935894 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.935902 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.935903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.936139 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.936335 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:40.936336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:40.936337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:40.936484 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:40.936708 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:40.936713 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:40.936890 (+ 177us) client_negotiation.cc:769] Sending connection context
0925 14:58:40.937001 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 14:58:40.937183 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260925 14:58:40.937892 16778 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44567
I20260925 14:58:40.937988 16778 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:40.938167 16778 heartbeater.cc:511] Master 127.8.13.254:44567 requested a full tablet report, sending...
I20260925 14:58:40.938553 16453 ts_manager.cc:194] Registered new tserver with Master: 312022ce3a7f4652b8eedf480cb75a7b (127.8.13.194:45879)
I20260925 14:58:40.938915 16453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:48053
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:40.959028 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:41.037665 16783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:41.037875 16783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:41.037907 16783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:41.039078 16783 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:41.039124 16783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:41.039157 16783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:41.040620 16783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16783
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:41.040865 16783 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:41.041077 16783 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:41.045825 16791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.045857 16789 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.045893 16788 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.046088 16783 server_base.cc:1034] running on GCE node
I20260925 14:58:41.046336 16783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:41.046561 16783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:41.047698 16783 hybrid_clock.cc:648] HybridClock initialized: now 1790348321047681 us; error 28 us; skew 500 ppm
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:41.050205 16783 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:41.050623 16783 webserver.cc:460] Webserver started at http://127.8.13.195:41035/ using document root <none> and password file <none>
I20260925 14:58:41.050894 16783 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:41.050942 16783 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:41.051069 16783 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:41.052158 16783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "2e1a1d4eff814208be02f5025651778d"
format_stamp: "Formatted at 2026-09-25 14:58:41 on dist-test-slave-15hz"
server_key: "7aeaad816123d8da51a01e8019e62344"
server_key_iv: "66e92522718af32825d46bcf1d845f6d"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.052516 16783 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1a1d4eff814208be02f5025651778d"
format_stamp: "Formatted at 2026-09-25 14:58:41 on dist-test-slave-15hz"
server_key: "7aeaad816123d8da51a01e8019e62344"
server_key_iv: "66e92522718af32825d46bcf1d845f6d"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.053820 16783 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:41.054572 16798 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.054744 16783 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:41.054818 16783 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "2e1a1d4eff814208be02f5025651778d"
format_stamp: "Formatted at 2026-09-25 14:58:41 on dist-test-slave-15hz"
server_key: "7aeaad816123d8da51a01e8019e62344"
server_key_iv: "66e92522718af32825d46bcf1d845f6d"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.054881 16783 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:41.063205 16783 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:41.065198 16783 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:41.065312 16783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:41.065500 16783 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:41.065771 16783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:41.065804 16783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.065824 16783 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:41.065834 16783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.071172 16783 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:34749
I20260925 14:58:41.071218 16911 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:34749 every 8 connection(s)
I20260925 14:58:41.071512 16783 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:41.073503 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16783
I20260925 14:58:41.073603 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:41.073782 8247 external_mini_cluster.cc:1428] Setting key 50c087ab4b09f2f07b8a34aa33cc096e
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:41.080667 16511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.072287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:51745 (local address 127.8.13.254:44567)
0925 14:58:41.072351 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.072353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.072543 (+ 190us) server_negotiation.cc:407] Connection header received
0925 14:58:41.073029 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.073030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.073040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.073098 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.074216 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.074861 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.075991 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.076192 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.076250 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.078276 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:41.078278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:41.078278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:41.078286 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:41.079471 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.079895 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.079897 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.079898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.079959 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.080163 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.080163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.080164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.080266 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:41.080313 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:41.080526 (+ 213us) server_negotiation.cc:299] Negotiation successful
0925 14:58:41.080593 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 14:58:41.080847 16914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.072256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:51745)
0925 14:58:41.072410 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.072452 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.072942 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.073158 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.073161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.073778 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.074124 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.074132 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.074967 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.074968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.075897 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.075900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.076260 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.076261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.076361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.076822 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.076831 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.078189 (+ 1358us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:41.079582 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:41.079591 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:41.079593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:41.079810 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:41.080012 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:41.080013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:41.080013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:41.080103 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:41.080308 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:41.080311 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:41.080425 (+ 114us) client_negotiation.cc:769] Sending connection context
0925 14:58:41.080502 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 14:58:41.080628 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260925 14:58:41.081305 16912 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44567
I20260925 14:58:41.081396 16912 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:41.081547 16912 heartbeater.cc:511] Master 127.8.13.254:44567 requested a full tablet report, sending...
I20260925 14:58:41.081892 16453 ts_manager.cc:194] Registered new tserver with Master: 2e1a1d4eff814208be02f5025651778d (127.8.13.195:34749)
I20260925 14:58:41.082232 16453 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:51745
I20260925 14:58:41.085265 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
Entry for principal oryx/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab.
WARNING: no policy specified for oryx/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
Entry for principal oryx/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab.
WARNING: no policy specified for oryx/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
Entry for principal oryx/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab.
I20260925 14:58:41.158844 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16515
I20260925 14:58:41.162838 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16649
I20260925 14:58:41.166664 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16783
I20260925 14:58:41.170008 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16421
I20260925 14:58:41.174315 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44567
--webserver_interface=127.8.13.254
--webserver_port=32835
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:41.254735 16921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:41.254961 16921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:41.254992 16921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:41.256223 16921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:41.256265 16921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:41.256284 16921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:41.256299 16921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:41.256314 16921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:41.257807 16921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=32835
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:41.258051 16921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:41.258262 16921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:41.263121 16926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.263074 16927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.263074 16929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.263141 16921 server_base.cc:1034] running on GCE node
I20260925 14:58:41.263584 16921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:41.263899 16921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:41.265077 16921 hybrid_clock.cc:648] HybridClock initialized: now 1790348321265073 us; error 46 us; skew 500 ppm
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:41.268070 16921 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:41.268644 16921 webserver.cc:460] Webserver started at http://127.8.13.254:32835/ using document root <none> and password file <none>
I20260925 14:58:41.268930 16921 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:41.269011 16921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:41.270450 16921 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260925 14:58:41.271303 16936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.271505 16921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:41.271617 16921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "16bd47926e024c138c9be2bd9d346fdd"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "779defbd4d6fe070540e5f60042580e4"
server_key_iv: "372f9ab603f603c017ce72338329d820"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.272075 16921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:41.286653 16921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:41.288695 16921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:41.288856 16921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:41.292511 16921 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44567
I20260925 14:58:41.292557 16988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44567 every 8 connection(s)
I20260925 14:58:41.292922 16921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:41.294407 16989 sys_catalog.cc:263] Verifying existing consensus state
I20260925 14:58:41.295079 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Bootstrap starting.
I20260925 14:58:41.299065 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16921
I20260925 14:58:41.299372 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:38613
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:44567
--webserver_port=41887
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 14:58:41.302783 16989 log.cc:824] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:41.304819 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 14:58:41.305078 16989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Bootstrap complete.
I20260925 14:58:41.307170 16989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:41.307399 16989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16bd47926e024c138c9be2bd9d346fdd, State: Initialized, Role: FOLLOWER
I20260925 14:58:41.307575 16989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:41.307654 16989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:41.307704 16989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:41.307756 16989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 1 FOLLOWER]: Advancing to term 2
I20260925 14:58:41.308756 16989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:41.308849 16989 leader_election.cc:302] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16bd47926e024c138c9be2bd9d346fdd; no voters:
I20260925 14:58:41.309023 16989 leader_election.cc:288] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 14:58:41.309068 16993 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 2 FOLLOWER]: Leader election won for term 2
I20260925 14:58:41.309180 16993 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [term 2 LEADER]: Becoming Leader. State: Replica: 16bd47926e024c138c9be2bd9d346fdd, State: Running, Role: LEADER
I20260925 14:58:41.309267 16993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } }
I20260925 14:58:41.309293 16989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:41.309571 16993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16bd47926e024c138c9be2bd9d346fdd. Latest consensus state: current_term: 2 leader_uuid: "16bd47926e024c138c9be2bd9d346fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } } }
I20260925 14:58:41.309646 16993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:41.309576 16994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16bd47926e024c138c9be2bd9d346fdd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16bd47926e024c138c9be2bd9d346fdd" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44567 } } }
I20260925 14:58:41.309854 16994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:41.311095 17009 catalog_manager.cc:1259] Loaded cluster ID: 80ba7e2b5c39431b845e1ef03dcb654d
I20260925 14:58:41.311149 17009 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: loading cluster ID for follower catalog manager: success
I20260925 14:58:41.312440 17009 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: acquiring CA information for follower catalog manager: success
I20260925 14:58:41.312786 17009 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 14:58:41.312858 16997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:41.313021 16997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:41.313079 16997 catalog_manager.cc:1259] Loaded cluster ID: 80ba7e2b5c39431b845e1ef03dcb654d
I20260925 14:58:41.313104 16997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:41.313282 16997 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:41.313359 16997 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16bd47926e024c138c9be2bd9d346fdd: Loaded TSK: 0
I20260925 14:58:41.313483 16997 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 14:58:41.386070 16991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:41.386302 16991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:41.386329 16991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:41.387631 16991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:41.387679 16991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:41.387709 16991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:41.389273 16991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.193.keytab
--principal=oryx/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:38613
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=41887
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:41.389511 16991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:41.389750 16991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:41.394603 17015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.394644 17016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.394726 16991 server_base.cc:1034] running on GCE node
W20260925 14:58:41.394622 17018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.395147 16991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:41.395387 16991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:41.396540 16991 hybrid_clock.cc:648] HybridClock initialized: now 1790348321396523 us; error 31 us; skew 500 ppm
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:41.399091 16991 init.cc:377] Logged in from keytab as oryx/127.8.13.193@KRBTEST.COM (short username oryx)
I20260925 14:58:41.399582 16991 webserver.cc:460] Webserver started at http://127.8.13.193:41887/ using document root <none> and password file <none>
I20260925 14:58:41.399816 16991 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:41.399878 16991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:41.401204 16991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:41.401751 17025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.401907 16991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:41.402012 16991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "454ba67e8fee499c9bd7964826408b40"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "ef0b156ef12a67464fdd6161617ef2cb"
server_key_iv: "5a5485d27036dbbd663e3991efffc603"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.402467 16991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:41.414438 16991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:41.416981 16991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:41.417145 16991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:41.417418 16991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:41.417781 16991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:41.417831 16991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.417862 16991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:41.417886 16991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.425132 16991 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:38613
I20260925 14:58:41.425158 17138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:38613 every 8 connection(s)
I20260925 14:58:41.425546 16991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.434072 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.426296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:57147)
0925 14:58:41.426471 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.426523 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.427031 (+ 508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.427458 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.427461 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.428065 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.428263 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.428267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.429395 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.429396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.430207 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.430209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.430804 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.430805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.430925 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.431463 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.431472 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.433856 (+ 2384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":84,"threads_started":1}
W20260925 14:58:41.434158 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.426350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57147 (local address 127.8.13.254:44567)
0925 14:58:41.426545 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.426553 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.426619 (+ 66us) server_negotiation.cc:407] Connection header received
0925 14:58:41.427241 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.427247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.427293 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.427409 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.428390 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.429297 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.430293 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.430724 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.430820 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.433993 (+ 3173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:57147: BlockingRecv error: recv got EOF from 127.8.13.193:57147 (error 108)
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":76,"threads_started":1}
W20260925 14:58:41.434326 17139 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:44567 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.254:44567: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
I20260925 14:58:41.434453 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 16991
I20260925 14:58:41.434784 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:45879
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:44567
--webserver_port=36433
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.439651 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.434540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:35957)
0925 14:58:41.434753 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.434761 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.434808 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.434980 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.434981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.435097 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.435149 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.435150 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.435807 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.435808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.436495 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.436497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.436985 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.436986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.437045 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.437482 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.437487 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.439569 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199}
W20260925 14:58:41.439738 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.434625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:35957 (local address 127.8.13.254:44567)
0925 14:58:41.434676 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.434678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.434807 (+ 129us) server_negotiation.cc:407] Connection header received
0925 14:58:41.434884 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.434885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.434901 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.434943 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.435218 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.435739 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.436585 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.436907 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.436981 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.439670 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:35957: BlockingRecv error: recv got EOF from 127.8.13.193:35957 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.445181 17141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.440315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:54383)
0925 14:58:41.440418 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.440431 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.440485 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.440637 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.440638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.440772 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.440813 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.440814 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.441694 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.441695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.442410 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.442413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.442846 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.442847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.442898 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.443372 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.443376 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.445120 (+ 1744us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":85}
W20260925 14:58:41.445286 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.440375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:54383 (local address 127.8.13.254:44567)
0925 14:58:41.440434 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.440436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.440470 (+ 34us) server_negotiation.cc:407] Connection header received
0925 14:58:41.440547 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.440548 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.440561 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.440619 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.440880 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.441613 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.442493 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.442772 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.442841 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.445223 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:54383: BlockingRecv error: recv got EOF from 127.8.13.193:54383 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260925 14:58:41.445324 17139 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:58:41.516839 17143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:41.517064 17143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:41.517088 17143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:41.518276 17143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:41.518321 17143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:41.518347 17143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:41.519788 17143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.194.keytab
--principal=oryx/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:45879
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=36433
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:41.520010 17143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:41.520222 17143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:41.524942 17149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.524935 17151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.524935 17148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.525189 17143 server_base.cc:1034] running on GCE node
I20260925 14:58:41.525403 17143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:41.525619 17143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:41.526762 17143 hybrid_clock.cc:648] HybridClock initialized: now 1790348321526746 us; error 27 us; skew 500 ppm
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:41.529162 17143 init.cc:377] Logged in from keytab as oryx/127.8.13.194@KRBTEST.COM (short username oryx)
I20260925 14:58:41.529569 17143 webserver.cc:460] Webserver started at http://127.8.13.194:36433/ using document root <none> and password file <none>
I20260925 14:58:41.529786 17143 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:41.529847 17143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:41.531142 17143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:41.531677 17158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.531816 17143 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:41.531903 17143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "312022ce3a7f4652b8eedf480cb75a7b"
format_stamp: "Formatted at 2026-09-25 14:58:40 on dist-test-slave-15hz"
server_key: "423037ca838a76de444f5295c93633fb"
server_key_iv: "bd3760c9384fbb18711bdc62623cb82e"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.532250 17143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:41.559443 17143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:41.561717 17143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:41.561882 17143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:41.562131 17143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:41.562427 17143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:41.562459 17143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.562489 17143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:41.562518 17143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.567891 17143 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45879
I20260925 14:58:41.567930 17271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45879 every 8 connection(s)
I20260925 14:58:41.568274 17143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:41.570338 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17143
I20260925 14:58:41.570605 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:34749
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:44567
--webserver_port=41035
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:35933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:58:41.576269 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.568950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56819 (local address 127.8.13.254:44567)
0925 14:58:41.569014 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.569016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.569230 (+ 214us) server_negotiation.cc:407] Connection header received
0925 14:58:41.569693 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.569694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.569708 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.569758 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.570802 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.571480 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.572517 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.572710 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.572768 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.576214 (+ 3446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:56819: BlockingRecv error: recv got EOF from 127.8.13.194:56819 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:41.576615 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.568940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:56819)
0925 14:58:41.569093 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.569129 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.569612 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.569829 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.569832 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.570426 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.570676 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.570683 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.571607 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.571608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.572424 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.572427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.572781 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.572782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.572923 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.573520 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.573530 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.576429 (+ 2899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
W20260925 14:58:41.576840 17272 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:44567 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.254:44567: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.581346 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.577074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:44895)
0925 14:58:41.577200 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.577212 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.577271 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.577386 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.577387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.577522 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.577586 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.577588 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.578217 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.578217 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.578733 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.578735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.579062 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.579062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.579127 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.579600 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.579604 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.581267 (+ 1663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260925 14:58:41.581411 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.577128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:44895 (local address 127.8.13.254:44567)
0925 14:58:41.577205 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.577207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.577253 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:58:41.577324 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.577325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.577335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.577382 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.577653 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.578146 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.578804 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.579000 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.579052 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.581352 (+ 2300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:44895: BlockingRecv error: recv got EOF from 127.8.13.194:44895 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.586254 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.581648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:34979)
0925 14:58:41.581693 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.581704 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.581747 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.581864 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.581865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.581999 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.582037 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.582038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.582612 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.582613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.583458 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.583461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.583891 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.583892 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.583948 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.584368 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.584372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.586181 (+ 1809us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260925 14:58:41.586395 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.581694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:34979 (local address 127.8.13.254:44567)
0925 14:58:41.581726 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.581728 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.581735 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:41.581802 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.581802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.581811 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.581859 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.582090 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.582554 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.583623 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.583822 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.583887 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.586274 (+ 2387us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:34979: BlockingRecv error: recv got EOF from 127.8.13.194:34979 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260925 14:58:41.586395 17272 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 14:58:41.651844 17275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:41.652060 17275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:41.652091 17275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:41.653290 17275 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:41.653332 17275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:41.653365 17275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:41.654807 17275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:35933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/krb5kdc/oryx_127.8.13.195.keytab
--principal=oryx/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=41035
--tserver_master_addrs=127.8.13.254:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17275
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:41.655050 17275 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:41.655269 17275 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:41.660081 17281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.660113 17280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:41.660171 17283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:41.660446 17275 server_base.cc:1034] running on GCE node
I20260925 14:58:41.660638 17275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:41.660840 17275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:41.661993 17275 hybrid_clock.cc:648] HybridClock initialized: now 1790348321661975 us; error 27 us; skew 500 ppm
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348321, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:41.664631 17275 init.cc:377] Logged in from keytab as oryx/127.8.13.195@KRBTEST.COM (short username oryx)
I20260925 14:58:41.665081 17275 webserver.cc:460] Webserver started at http://127.8.13.195:41035/ using document root <none> and password file <none>
I20260925 14:58:41.665315 17275 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:41.665375 17275 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:41.666731 17275 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:41.667339 17290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.667486 17275 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:41.667604 17275 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "2e1a1d4eff814208be02f5025651778d"
format_stamp: "Formatted at 2026-09-25 14:58:41 on dist-test-slave-15hz"
server_key: "7aeaad816123d8da51a01e8019e62344"
server_key_iv: "66e92522718af32825d46bcf1d845f6d"
server_key_version: "encryptionkey@0"
I20260925 14:58:41.667938 17275 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:41.678824 17275 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:41.680804 17275 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:41.680939 17275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:41.681155 17275 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:41.681448 17275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:41.681480 17275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.681509 17275 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:41.681526 17275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:41.686767 17275 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:34749
I20260925 14:58:41.686825 17403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:34749 every 8 connection(s)
I20260925 14:58:41.687141 17275 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
W20260925 14:58:41.693951 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.687848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:38995 (local address 127.8.13.254:44567)
0925 14:58:41.687927 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.687929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.688098 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:58:41.688562 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.688563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.688573 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.688628 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.689496 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.689970 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.690874 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.691044 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.691085 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.693901 (+ 2816us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:38995: BlockingRecv error: recv got EOF from 127.8.13.195:38995 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:58:41.693969 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.687826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:38995)
0925 14:58:41.687970 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.688014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.688485 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.688690 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.688693 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.689235 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.689426 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.689431 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.690028 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.690029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.690813 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.690815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.691099 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.691100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.691197 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.691668 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.691676 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.693815 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
W20260925 14:58:41.694203 17404 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:44567 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.13.254:44567: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
I20260925 14:58:41.695751 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17275
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.698849 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.694416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:53185)
0925 14:58:41.694463 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.694474 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.694512 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.694734 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.694734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.694841 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.694891 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.694893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.695509 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.695509 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.696160 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.696163 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.696585 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.696586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.696654 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.697030 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.697035 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.698777 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 14:58:41.698899 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.694587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:53185 (local address 127.8.13.254:44567)
0925 14:58:41.694639 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.694641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.694648 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:41.694669 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.694670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.694678 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.694722 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.694942 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.695438 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.696237 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.696517 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.696580 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.698848 (+ 2268us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:53185: BlockingRecv error: recv got EOF from 127.8.13.195:53185 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:41 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:41.703315 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.699164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:57457)
0925 14:58:41.699214 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:41.699223 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:41.699251 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:41.699364 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:41.699365 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:41.699492 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:41.699563 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.699565 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.700395 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.700396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.700886 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:41.700887 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.701209 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.701210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:41.701266 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.701537 (+ 271us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:41.701542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:41.703249 (+ 1707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 14:58:41.703374 17142 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:41.699206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:57457 (local address 127.8.13.254:44567)
0925 14:58:41.699243 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.699245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.699252 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:41.699304 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.699305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.699313 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.699361 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.699619 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.700303 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.700951 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.701117 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.701173 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.703324 (+ 2151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:57457: BlockingRecv error: recv got EOF from 127.8.13.195:57457 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 14:58:41.703501 17404 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 14:58:41.703994 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:41.696151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48340 (local address 127.8.13.254:44567)
0925 14:58:41.696255 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:41.696257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:41.696403 (+ 146us) server_negotiation.cc:407] Connection header received
0925 14:58:41.696469 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:41.696470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:41.696480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:41.696526 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:41.697097 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.697733 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.698722 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:41.698997 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:41.699060 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:41.700995 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:41.701000 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:41.701000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:41.701012 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:41.702560 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.702909 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.702912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.702913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.703042 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:41.703259 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:41.703260 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:41.703260 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:41.703643 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:41.703751 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:41.703845 (+ 94us) server_negotiation.cc:299] Negotiation successful
0925 14:58:41.703926 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":57,"threads_started":1}
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:42.450407 17409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.445673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:60987)
0925 14:58:42.445809 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.445822 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:42.445895 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:42.446104 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:42.446105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:42.446280 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:42.446331 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.446333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.447069 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.447070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.447706 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.447709 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.448083 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.448083 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.448130 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.448564 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:42.448568 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:42.450330 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":62,"threads_started":1}
W20260925 14:58:42.450482 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:42.445729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:60987 (local address 127.8.13.254:44567)
0925 14:58:42.445817 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:42.445819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:42.445889 (+ 70us) server_negotiation.cc:407] Connection header received
0925 14:58:42.445948 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:42.445949 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:42.445959 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:42.446007 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:42.446458 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.446967 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.447791 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.447995 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.448069 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.450422 (+ 2353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:60987: BlockingRecv error: recv got EOF from 127.8.13.193:60987 (error 108)
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":44,"threads_started":1}
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:42.590937 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.586756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:39425)
0925 14:58:42.586877 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.586887 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:42.586947 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:42.587056 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:42.587056 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:42.587216 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:42.587267 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.587269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.587916 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.587917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.588475 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.588477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.588799 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.588800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.588836 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.589288 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:42.589293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:42.590865 (+ 1572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
W20260925 14:58:42.591006 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:42.586813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:39425 (local address 127.8.13.254:44567)
0925 14:58:42.586866 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:42.586868 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:42.586940 (+ 72us) server_negotiation.cc:407] Connection header received
0925 14:58:42.586984 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:42.586985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:42.586996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:42.587046 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:42.587318 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.587853 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.588556 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.588745 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.588798 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.590953 (+ 2155us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:39425: BlockingRecv error: recv got EOF from 127.8.13.194:39425 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:42 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:42.708635 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:42.703955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:43727)
0925 14:58:42.704117 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:42.704141 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:42.704218 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:42.704355 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:42.704356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:42.704522 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:42.704570 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.704572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.705241 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.705242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.705839 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:42.705842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.706191 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.706192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:42.706245 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.706716 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:42.706721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:42.708543 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":98,"threads_started":1}
W20260925 14:58:42.708710 17410 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:42.704059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:43727 (local address 127.8.13.254:44567)
0925 14:58:42.704127 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:42.704129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:42.704204 (+ 75us) server_negotiation.cc:407] Connection header received
0925 14:58:42.704280 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:42.704281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:42.704296 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:42.704343 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:42.704655 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.705161 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.705918 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:42.706124 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:42.706173 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:42.708658 (+ 2485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:43727: BlockingRecv error: recv got EOF from 127.8.13.195:43727 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:43.458007 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.450941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:58111)
0925 14:58:43.451077 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.451087 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:43.451157 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:43.451314 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:43.451315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:43.451468 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:43.451517 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.451519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.452165 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.452166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.452699 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.452701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.453072 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.453074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.453124 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.456187 (+ 3063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:43.456192 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:43.457938 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":60,"threads_started":1}
W20260925 14:58:43.458132 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:43.451069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:58111 (local address 127.8.13.254:44567)
0925 14:58:43.451191 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:43.451193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:43.451200 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:43.451225 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:43.451225 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:43.451236 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:43.451301 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:43.451608 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.452104 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.452781 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.452977 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.453032 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.458051 (+ 5019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:58111: BlockingRecv error: recv got EOF from 127.8.13.193:58111 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":60,"threads_started":1}
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:43.596379 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.591470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:48661)
0925 14:58:43.591627 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.591639 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:43.591713 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:43.591838 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:43.591838 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:43.592010 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:43.592056 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.592058 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.592696 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.592697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.593279 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.593281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.593770 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.593771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.593811 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.594266 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:43.594271 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:43.596284 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":81,"threads_started":1}
W20260925 14:58:43.596443 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:43.591503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:48661 (local address 127.8.13.254:44567)
0925 14:58:43.591567 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:43.591569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:43.591694 (+ 125us) server_negotiation.cc:407] Connection header received
0925 14:58:43.591764 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:43.591765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:43.591778 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:43.591827 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:43.592131 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.592625 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.593368 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.593693 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.593776 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.596377 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:48661: BlockingRecv error: recv got EOF from 127.8.13.194:48661 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:43 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:43.713889 17416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:43.709178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:53843)
0925 14:58:43.709366 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:43.709379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:43.709442 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:43.709646 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:43.709647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:43.709845 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:43.709894 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.709896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.710641 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.710642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.711199 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:43.711201 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.711668 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.711669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:43.711717 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.712163 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:43.712166 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:43.713820 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":93,"threads_started":1}
W20260925 14:58:43.714018 17414 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:43.709338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:53843 (local address 127.8.13.254:44567)
0925 14:58:43.709425 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:43.709427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:43.709438 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:43.709519 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:43.709520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:43.709534 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:43.709596 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:43.710011 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.710528 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.711302 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:43.711498 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:43.711584 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:43.713942 (+ 2358us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:53843: BlockingRecv error: recv got EOF from 127.8.13.195:53843 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:44.462862 17417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.458461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:37435)
0925 14:58:44.458587 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.458597 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.458656 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.458787 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.458788 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.458945 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.458996 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.458998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.459656 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.459657 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.460205 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.460207 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.460568 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.460569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.460603 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.461052 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.461058 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.462788 (+ 1730us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":72,"threads_started":1}
W20260925 14:58:44.462994 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:44.458566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:37435 (local address 127.8.13.254:44567)
0925 14:58:44.458677 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.458680 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.458689 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:44.458710 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.458711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.458720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.458777 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.459061 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.459590 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.460291 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.460512 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.460565 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.462908 (+ 2343us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:37435: BlockingRecv error: recv got EOF from 127.8.13.193:37435 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":44,"threads_started":1}
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:44.601399 17419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.596979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:42365)
0925 14:58:44.597152 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.597163 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.597224 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.597377 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.597377 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.597525 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.597576 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.597578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.598216 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.598217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.598762 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.598764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.599193 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.599194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.599256 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.599733 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.599737 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.601329 (+ 1592us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
W20260925 14:58:44.601485 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:44.597111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:42365 (local address 127.8.13.254:44567)
0925 14:58:44.597171 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.597172 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.597211 (+ 39us) server_negotiation.cc:407] Connection header received
0925 14:58:44.597299 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.597300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.597313 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.597362 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.597628 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.598140 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.598871 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.599113 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.599183 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.601423 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:42365: BlockingRecv error: recv got EOF from 127.8.13.194:42365 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:44 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:44.718600 17420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:44.714380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:60269)
0925 14:58:44.714498 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:44.714509 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:44.714560 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:44.714676 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:44.714677 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:44.714828 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:44.714874 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.714875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.715554 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.715555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.716104 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:44.716106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.716479 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.716480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:44.716516 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.716925 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:44.716929 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:44.718533 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
W20260925 14:58:44.718680 17418 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:44.714481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:60269 (local address 127.8.13.254:44567)
0925 14:58:44.714535 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:44.714537 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:44.714564 (+ 27us) server_negotiation.cc:407] Connection header received
0925 14:58:44.714617 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:44.714618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:44.714631 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:44.714668 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:44.714941 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.715454 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.716215 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:44.716411 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:44.716470 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:44.718616 (+ 2146us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:60269: BlockingRecv error: recv got EOF from 127.8.13.195:60269 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:45.468029 17421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.463508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:47241)
0925 14:58:45.463689 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.463699 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:45.463753 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:45.463948 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:45.463949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:45.464148 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:45.464212 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.464214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.464912 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.464913 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.465461 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.465463 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.465768 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.465769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.465807 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.466284 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:45.466289 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:45.467950 (+ 1661us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
W20260925 14:58:45.468106 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:45.463614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:47241 (local address 127.8.13.254:44567)
0925 14:58:45.463739 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:45.463741 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:45.463755 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:58:45.463821 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:45.463821 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:45.463832 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:45.463931 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:45.464286 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.464817 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.465533 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.465719 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.465761 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.468048 (+ 2287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:47241: BlockingRecv error: recv got EOF from 127.8.13.193:47241 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:45.606467 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.601966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:37865)
0925 14:58:45.602141 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.602151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:45.602211 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:45.602340 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:45.602340 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:45.602491 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:45.602545 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.602547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.603135 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.603136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.603685 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.603686 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.603994 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.603995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.604046 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.604476 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:45.604481 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:45.606405 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
W20260925 14:58:45.606555 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:45.602082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:37865 (local address 127.8.13.254:44567)
0925 14:58:45.602144 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:45.602146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:45.602205 (+ 59us) server_negotiation.cc:407] Connection header received
0925 14:58:45.602255 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:45.602256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:45.602267 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:45.602329 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:45.602593 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.603077 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.603755 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.603939 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.603982 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.606498 (+ 2516us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:37865: BlockingRecv error: recv got EOF from 127.8.13.194:37865 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:45 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:45.723757 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:45.719070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:35935)
0925 14:58:45.719219 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:45.719231 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:45.719300 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:45.719435 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:45.719436 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:45.719636 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:45.719686 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.719689 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.720335 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.720337 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.720862 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:45.720865 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.721186 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.721187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:45.721237 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.721700 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:45.721704 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:45.723666 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
W20260925 14:58:45.723850 17422 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:45.719186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:35935 (local address 127.8.13.254:44567)
0925 14:58:45.719251 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:45.719254 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:45.719282 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:58:45.719357 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:45.719358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:45.719373 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:45.719420 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:45.719763 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.720260 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.720938 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:45.721120 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:45.721178 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:45.723785 (+ 2607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:35935: BlockingRecv error: recv got EOF from 127.8.13.195:35935 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:46.476257 17425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.468581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:36699)
0925 14:58:46.468754 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.468767 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:46.468852 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:46.468988 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:46.468988 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:46.469162 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:46.469214 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.469216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.469870 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.469871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.470505 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.470508 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.470882 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.470883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.470948 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.474351 (+ 3403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:46.474357 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:46.476193 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":78,"threads_started":1}
W20260925 14:58:46.476366 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:46.468674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:36699 (local address 127.8.13.254:44567)
0925 14:58:46.468779 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:46.468781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:46.468839 (+ 58us) server_negotiation.cc:407] Connection header received
0925 14:58:46.468908 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:46.468909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:46.468920 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:46.468979 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:46.469279 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.469778 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.470596 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.470805 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.470856 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.476298 (+ 5442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:36699: BlockingRecv error: recv got EOF from 127.8.13.193:36699 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":49,"threads_started":1}
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:46.611860 17427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.606941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:52601)
0925 14:58:46.607074 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.607087 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:46.607155 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:46.607291 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:46.607291 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:46.607596 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:46.607658 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.607662 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.608410 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.608411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.609065 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.609068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.609455 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.609456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.609512 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.610060 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:46.610065 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:46.611776 (+ 1711us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":64,"threads_started":1}
W20260925 14:58:46.612176 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:46.607050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:52601 (local address 127.8.13.254:44567)
0925 14:58:46.607108 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:46.607110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:46.607136 (+ 26us) server_negotiation.cc:407] Connection header received
0925 14:58:46.607221 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:46.607222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:46.607234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:46.607280 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:46.607793 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.608319 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.609155 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.609384 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.609455 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.611893 (+ 2438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:52601: BlockingRecv error: recv got EOF from 127.8.13.194:52601 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:46 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:46.728866 17428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:46.724297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:34445)
0925 14:58:46.724418 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:46.724430 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:46.724480 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:46.724608 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:46.724609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:46.724781 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:46.724830 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.724832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.725585 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.725587 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.726150 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:46.726152 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.726536 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.726537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:46.726594 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.727076 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:46.727080 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:46.728788 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260925 14:58:46.728950 17426 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:46.724410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:34445 (local address 127.8.13.254:44567)
0925 14:58:46.724489 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:46.724490 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:46.724500 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:46.724524 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:46.724525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:46.724537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:46.724594 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:46.724909 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.725447 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.726245 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:46.726442 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:46.726510 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:46.728895 (+ 2385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:34445: BlockingRecv error: recv got EOF from 127.8.13.195:34445 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:47.481272 17429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.476765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:52369)
0925 14:58:47.476899 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.476911 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.476973 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.477140 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.477141 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.477316 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.477367 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.477369 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.478019 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.478020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.478605 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.478606 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.478997 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.478997 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.479035 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.479467 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:47.479472 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.481214 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":63,"threads_started":1}
W20260925 14:58:47.481371 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:47.476822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:52369 (local address 127.8.13.254:44567)
0925 14:58:47.476947 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.476949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.476959 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:47.477029 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.477029 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.477040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.477086 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.477437 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.477941 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.478699 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.478900 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.478997 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.481306 (+ 2309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:52369: BlockingRecv error: recv got EOF from 127.8.13.193:52369 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":71,"threads_started":1}
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:47.616801 17431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.612361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:60403)
0925 14:58:47.612482 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.612494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.612559 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.612684 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.612685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.612854 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.612902 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.612904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.613551 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.613552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.614095 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.614096 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.614429 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.614429 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.614483 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.614930 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:47.614935 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.616721 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":68,"threads_started":1}
W20260925 14:58:47.616888 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:47.612432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:60403 (local address 127.8.13.254:44567)
0925 14:58:47.612527 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.612528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.612540 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:58:47.612623 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.612623 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.612634 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.612679 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.612976 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.613477 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.614179 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.614371 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.614427 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.616836 (+ 2409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:60403: BlockingRecv error: recv got EOF from 127.8.13.194:60403 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:47 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:47.734179 17432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:47.729438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:45273)
0925 14:58:47.729593 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:47.729603 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:47.729667 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:47.729822 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:47.729822 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:47.730055 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:47.730109 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.730112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.730790 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.730791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.731491 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:47.731494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.731903 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.731904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:47.731942 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.732381 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:47.732385 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:47.734101 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":67,"threads_started":1}
W20260925 14:58:47.734249 17430 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:47.729535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:45273 (local address 127.8.13.254:44567)
0925 14:58:47.729584 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:47.729585 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:47.729652 (+ 67us) server_negotiation.cc:407] Connection header received
0925 14:58:47.729736 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:47.729737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:47.729751 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:47.729799 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:47.730177 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.730682 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.731623 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:47.731840 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:47.731901 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:47.734198 (+ 2297us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:45273: BlockingRecv error: recv got EOF from 127.8.13.195:45273 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:48.485996 17433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.481732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:47815)
0925 14:58:48.481859 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.481869 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:48.481919 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:48.482046 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:48.482047 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:48.482204 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:48.482252 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.482254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.482886 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.482887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.483422 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.483425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.483772 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.483773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.483811 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.484287 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:48.484292 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:48.485934 (+ 1642us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":66,"threads_started":1}
W20260925 14:58:48.486119 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:48.481802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:47815 (local address 127.8.13.254:44567)
0925 14:58:48.481921 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:48.481923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:48.481933 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:48.481957 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:48.481957 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:48.481968 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:48.482032 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:48.482306 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.482798 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.483496 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.483719 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.483763 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.486032 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:47815: BlockingRecv error: recv got EOF from 127.8.13.193:47815 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":54,"threads_started":1}
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:48.621371 17435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.617286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:32901)
0925 14:58:48.617421 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.617432 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:48.617490 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:48.617597 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:48.617598 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:48.617745 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:48.617792 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.617794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.618372 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.618373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.618882 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.618884 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.619163 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.619163 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.619198 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.619677 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:48.619682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:48.621294 (+ 1612us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":72,"threads_started":1}
W20260925 14:58:48.621469 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:48.617333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:32901 (local address 127.8.13.254:44567)
0925 14:58:48.617434 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:48.617436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:48.617487 (+ 51us) server_negotiation.cc:407] Connection header received
0925 14:58:48.617543 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:48.617544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:48.617555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:48.617592 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:48.617838 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.618320 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.618939 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.619113 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.619155 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.621415 (+ 2260us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:32901: BlockingRecv error: recv got EOF from 127.8.13.194:32901 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:48 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:48.739033 17436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:48.734622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:57305)
0925 14:58:48.734775 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:48.734787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:48.734850 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:48.734975 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:48.734976 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:48.735164 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:48.735211 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.735213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.735939 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.735940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.736480 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:48.736482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.736815 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.736816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:48.736869 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.737295 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:48.737299 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:48.738972 (+ 1673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
W20260925 14:58:48.739130 17434 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:48.734687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:57305 (local address 127.8.13.254:44567)
0925 14:58:48.734768 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:48.734770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:48.734840 (+ 70us) server_negotiation.cc:407] Connection header received
0925 14:58:48.734914 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:48.734915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:48.734929 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:48.734970 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:48.735283 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.735860 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.736559 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:48.736757 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:48.736810 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:48.739062 (+ 2252us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:57305: BlockingRecv error: recv got EOF from 127.8.13.195:57305 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:49.491127 17437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.486559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:52701)
0925 14:58:49.486709 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.486719 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:49.486767 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:49.486933 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:49.486934 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:49.487178 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:49.487230 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.487232 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.487944 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.487945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.488498 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.488499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.488908 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.488909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.488955 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.489371 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:49.489374 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:49.491066 (+ 1692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":66,"threads_started":1}
W20260925 14:58:49.491231 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:49.486708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:52701 (local address 127.8.13.254:44567)
0925 14:58:49.486821 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.486823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.486831 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:49.486853 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.486854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.486863 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.486922 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:49.487300 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.487864 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.488598 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.488854 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.488901 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.491163 (+ 2262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:52701: BlockingRecv error: recv got EOF from 127.8.13.193:52701 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:49.626659 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.621891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:41907)
0925 14:58:49.622047 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.622058 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:49.622113 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:49.622240 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:49.622241 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:49.622399 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:49.622448 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.622449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.623303 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.623305 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.623903 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.623905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.624326 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.624327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.624375 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.624834 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:49.624839 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:49.626581 (+ 1742us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":69,"threads_started":1}
W20260925 14:58:49.626814 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:49.622002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:41907 (local address 127.8.13.254:44567)
0925 14:58:49.622058 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.622060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.622110 (+ 50us) server_negotiation.cc:407] Connection header received
0925 14:58:49.622173 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.622174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.622184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.622224 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:49.622509 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.623211 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.624023 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.624240 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.624302 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.626695 (+ 2393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:41907: BlockingRecv error: recv got EOF from 127.8.13.194:41907 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:49 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:49.743950 17440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:49.739656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:55435)
0925 14:58:49.739772 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:49.739781 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:49.739834 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:49.739961 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:49.739962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:49.740128 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:49.740177 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.740179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.740848 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.740849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.741375 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:49.741377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.741736 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.741738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:49.741787 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.742236 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:49.742240 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:49.743889 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
W20260925 14:58:49.744072 17438 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:49.739732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:55435 (local address 127.8.13.254:44567)
0925 14:58:49.739795 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:49.739797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:49.739821 (+ 24us) server_negotiation.cc:407] Connection header received
0925 14:58:49.739880 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:49.739881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:49.739894 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:49.739944 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:49.740268 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.740779 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.741456 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:49.741660 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:49.741713 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:49.743999 (+ 2286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:55435: BlockingRecv error: recv got EOF from 127.8.13.195:55435 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:50.496033 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.491606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:37971)
0925 14:58:50.491716 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.491729 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:50.491790 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:50.491991 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:50.491992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:50.492153 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:50.492202 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.492203 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.492810 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.492810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.493321 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.493323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.493723 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.493724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.493767 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.494246 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:50.494250 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:50.495974 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":54,"threads_started":1}
W20260925 14:58:50.496191 17442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:50.491675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:37971 (local address 127.8.13.254:44567)
0925 14:58:50.491844 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.491846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.491857 (+ 11us) server_negotiation.cc:407] Connection header received
0925 14:58:50.491882 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.491883 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.491893 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.491985 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:50.492256 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.492756 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.493413 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.493639 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.493730 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.496107 (+ 2377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:37971: BlockingRecv error: recv got EOF from 127.8.13.193:37971 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":73,"threads_started":1}
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:50.631888 17443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.627222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:45257)
0925 14:58:50.627359 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.627371 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:50.627435 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:50.627635 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:50.627637 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:50.627795 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:50.627845 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.627847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.628472 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.628472 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.629059 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.629062 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.629434 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.629435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.629502 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.629997 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:50.630001 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:50.631825 (+ 1824us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":69,"threads_started":1}
W20260925 14:58:50.632051 17442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:50.627327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:45257 (local address 127.8.13.254:44567)
0925 14:58:50.627403 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.627405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.627419 (+ 14us) server_negotiation.cc:407] Connection header received
0925 14:58:50.627492 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.627493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.627504 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.627581 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:50.627912 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.628413 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.629115 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.629358 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.629422 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.631982 (+ 2560us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:45257: BlockingRecv error: recv got EOF from 127.8.13.194:45257 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:50 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:50.749142 17444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:50.744538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:58371)
0925 14:58:50.744695 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:50.744708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:50.744790 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:50.744946 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:50.744947 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:50.745116 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:50.745163 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.745165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.745804 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.745805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.746324 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:50.746325 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.746647 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.746648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:50.746701 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.747144 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:50.747149 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:50.749082 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
W20260925 14:58:50.749241 17442 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:50.744644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:58371 (local address 127.8.13.254:44567)
0925 14:58:50.744731 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:50.744733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:50.744781 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:58:50.744860 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:50.744861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:50.744879 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:50.744926 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:50.745236 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.745728 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.746400 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:50.746583 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:50.746640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:50.749166 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:58371: BlockingRecv error: recv got EOF from 127.8.13.195:58371 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:51.503711 17445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.496516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:38515)
0925 14:58:51.496654 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.496665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:51.496715 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:51.496853 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:51.496853 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:51.497005 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:51.497054 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.497056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.497677 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.497678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.498206 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.498207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.498550 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.498551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.498627 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.501956 (+ 3329us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:51.501961 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:51.503650 (+ 1689us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":84,"threads_started":1}
W20260925 14:58:51.503839 17446 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:51.496603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:38515 (local address 127.8.13.254:44567)
0925 14:58:51.496712 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:51.496714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:51.496754 (+ 40us) server_negotiation.cc:407] Connection header received
0925 14:58:51.496778 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:51.496778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:51.496790 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:51.496838 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:51.497126 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.497621 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.498265 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.498481 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.498536 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.503760 (+ 5224us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:38515: BlockingRecv error: recv got EOF from 127.8.13.193:38515 (error 108)
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:51.636689 17447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.632322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:55055)
0925 14:58:51.632456 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.632468 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:51.632525 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:51.632652 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:51.632653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:51.632806 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:51.632852 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.632853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.633463 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.633464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.633996 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.633998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.634337 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.634338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.634383 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.634820 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:51.634824 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:51.636610 (+ 1786us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260925 14:58:51.636771 17446 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:51.632421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:55055 (local address 127.8.13.254:44567)
0925 14:58:51.632474 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:51.632475 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:51.632521 (+ 46us) server_negotiation.cc:407] Connection header received
0925 14:58:51.632569 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:51.632570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:51.632581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:51.632632 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:51.632899 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.633391 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.634075 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.634272 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.634325 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.636716 (+ 2391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:55055: BlockingRecv error: recv got EOF from 127.8.13.194:55055 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:51 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:51.753933 17448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:51.749645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:54357)
0925 14:58:51.749812 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:51.749824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:51.749875 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:51.750002 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:51.750003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:51.750162 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:51.750211 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.750212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.750795 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.750795 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.751327 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:51.751328 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.751679 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.751680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:51.751735 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.752157 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:51.752162 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:51.753864 (+ 1702us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":72,"threads_started":1}
W20260925 14:58:51.754001 17446 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:51.749780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:54357 (local address 127.8.13.254:44567)
0925 14:58:51.749856 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:51.749858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:51.749867 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:51.749925 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:51.749926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:51.749938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:51.749983 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:51.750262 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.750741 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.751393 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:51.751596 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:51.751644 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:51.753951 (+ 2307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:54357: BlockingRecv error: recv got EOF from 127.8.13.195:54357 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:52.508390 17449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.504152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:35949)
0925 14:58:52.504301 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.504312 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:52.504382 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:52.504517 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:52.504518 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:52.504694 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:52.504741 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.504742 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.505344 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.505345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.505912 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.505915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.506208 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.506209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.506252 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.506636 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:52.506640 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:52.508317 (+ 1677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
W20260925 14:58:52.508555 17450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:52.504226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:35949 (local address 127.8.13.254:44567)
0925 14:58:52.504325 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:52.504327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:52.504375 (+ 48us) server_negotiation.cc:407] Connection header received
0925 14:58:52.504433 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:52.504434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:52.504445 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:52.504502 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:52.504790 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.505283 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.505971 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.506151 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.506194 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.508412 (+ 2218us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:35949: BlockingRecv error: recv got EOF from 127.8.13.193:35949 (error 108)
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":44,"threads_started":1}
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:52.641788 17451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.637249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:46815)
0925 14:58:52.637390 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.637399 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:52.637455 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:52.637629 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:52.637629 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:52.637789 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:52.637834 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.637836 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.638660 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.638661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.639217 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.639219 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.639556 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.639556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.639594 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.640015 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:52.640019 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:52.641709 (+ 1690us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
W20260925 14:58:52.641877 17450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:52.637332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:46815 (local address 127.8.13.254:44567)
0925 14:58:52.637415 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:52.637417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:52.637446 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:58:52.637539 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:52.637541 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:52.637553 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:52.637618 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:52.637929 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.638581 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.639282 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.639474 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.639551 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.641816 (+ 2265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:46815: BlockingRecv error: recv got EOF from 127.8.13.194:46815 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:52 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:52.759022 17452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:52.754529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:33111)
0925 14:58:52.754716 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:52.754728 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:52.754807 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:52.754937 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:52.754938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:52.755114 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:52.755158 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.755160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.755892 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.755894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.756447 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:52.756448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.756820 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.756821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:52.756879 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.757289 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:52.757293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:52.758928 (+ 1635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":102,"threads_started":1}
W20260925 14:58:52.759099 17450 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:52.754618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:33111 (local address 127.8.13.254:44567)
0925 14:58:52.754720 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:52.754722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:52.754803 (+ 81us) server_negotiation.cc:407] Connection header received
0925 14:58:52.754864 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:52.754866 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:52.754880 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:52.754925 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:52.755249 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.755816 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.756533 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:52.756755 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:52.756811 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:52.759045 (+ 2234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:33111: BlockingRecv error: recv got EOF from 127.8.13.195:33111 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:53.514063 17453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.508803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:56015)
0925 14:58:53.508950 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.508962 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:53.509021 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:53.509189 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:53.509190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:53.509351 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:53.509399 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.509401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.510389 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.510390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.510936 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.510939 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.511401 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.511401 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.511449 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.511969 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:53.511973 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:53.513947 (+ 1974us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":64,"threads_started":1}
W20260925 14:58:53.514097 17454 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:53.508889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:56015 (local address 127.8.13.254:44567)
0925 14:58:53.509039 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:53.509042 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:53.509050 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:53.509086 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:53.509086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:53.509101 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:53.509179 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:53.509460 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.510312 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.511006 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.511328 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.511394 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.514041 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:56015: BlockingRecv error: recv got EOF from 127.8.13.193:56015 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":67,"threads_started":1}
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:53.646684 17455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.642255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:44761)
0925 14:58:53.642393 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.642405 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:53.642477 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:53.642622 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:53.642623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:53.642796 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:53.642838 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.642840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.643482 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.643483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.644072 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.644074 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.644393 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.644394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.644450 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.644894 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:53.644898 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:53.646595 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
W20260925 14:58:53.646746 17454 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:53.642343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:44761 (local address 127.8.13.254:44567)
0925 14:58:53.642398 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:53.642400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:53.642457 (+ 57us) server_negotiation.cc:407] Connection header received
0925 14:58:53.642539 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:53.642540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:53.642551 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:53.642613 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:53.642907 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.643404 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.644151 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.644336 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.644391 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.646695 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:44761: BlockingRecv error: recv got EOF from 127.8.13.194:44761 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:53 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:53.764503 17456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:53.759678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:55143)
0925 14:58:53.759838 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:53.759851 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:53.759910 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:53.760121 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:53.760122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:53.760295 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:53.760340 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.760341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.761059 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.761061 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.761622 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:53.761623 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.761998 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.761999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:53.762047 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.762557 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:53.762562 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:53.764423 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":72,"threads_started":1}
W20260925 14:58:53.764604 17454 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:53.759802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:55143 (local address 127.8.13.254:44567)
0925 14:58:53.759861 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:53.759863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:53.759898 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:58:53.759972 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:53.759973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:53.759987 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:53.760045 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:53.760441 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.760965 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.761717 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:53.761923 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:53.761978 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:53.764531 (+ 2553us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:55143: BlockingRecv error: recv got EOF from 127.8.13.195:55143 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:54.518910 17457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.514579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:58495)
0925 14:58:54.514716 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.514727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.514778 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.514905 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.514905 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.515065 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.515116 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.515118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.515793 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.515794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.516357 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.516359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.516710 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.516710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.516746 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.517231 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.517235 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.518840 (+ 1605us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":75,"threads_started":1}
W20260925 14:58:54.519013 17458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:54.514639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:58495 (local address 127.8.13.254:44567)
0925 14:58:54.514778 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.514780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.514810 (+ 30us) server_negotiation.cc:407] Connection header received
0925 14:58:54.514832 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.514833 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.514844 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.514896 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.515176 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.515735 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.516447 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.516648 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.516707 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.518934 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:58495: BlockingRecv error: recv got EOF from 127.8.13.193:58495 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:54.651381 17459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.647228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:54649)
0925 14:58:54.647370 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.647385 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.647462 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.647616 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.647616 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.647780 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.647828 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.647829 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.648426 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.648426 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.648970 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.648971 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.649254 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.649254 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.649292 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.649707 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.649711 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.651309 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":72,"threads_started":1}
W20260925 14:58:54.651468 17458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:54.647405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:54649 (local address 127.8.13.254:44567)
0925 14:58:54.647467 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.647469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.647479 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:54.647521 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.647522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.647550 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.647598 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.647881 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.648371 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.649030 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.649206 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.649247 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.651409 (+ 2162us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:54649: BlockingRecv error: recv got EOF from 127.8.13.194:54649 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:54 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:54.769351 17460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:54.765113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:45221)
0925 14:58:54.765232 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:54.765244 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:54.765307 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:54.765426 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:54.765427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:54.765590 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:54.765636 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.765637 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.766263 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.766263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.766810 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:54.766811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.767130 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.767130 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:54.767165 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.767633 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:54.767638 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:54.769279 (+ 1641us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
W20260925 14:58:54.769424 17458 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:54.765186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:45221 (local address 127.8.13.254:44567)
0925 14:58:54.765256 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:54.765258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:54.765294 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:58:54.765350 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:54.765351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:54.765364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:54.765414 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:54.765691 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.766181 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.766883 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:54.767072 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:54.767126 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:54.769372 (+ 2246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:45221: BlockingRecv error: recv got EOF from 127.8.13.195:45221 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:55.523993 17461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.519442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:57081)
0925 14:58:55.519607 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.519618 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:55.519669 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:55.519820 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:55.519821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:55.519984 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:55.520035 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.520036 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.520740 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.520742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.521300 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.521302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.521662 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.521663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.521700 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.522146 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:55.522150 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:55.523920 (+ 1770us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":61,"threads_started":1}
W20260925 14:58:55.524070 17462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.519581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57081 (local address 127.8.13.254:44567)
0925 14:58:55.519692 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.519694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.519702 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:55.519728 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.519729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.519738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.519808 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.520132 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.520665 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.521389 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.521601 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.521660 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.524005 (+ 2345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:57081: BlockingRecv error: recv got EOF from 127.8.13.193:57081 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":51,"threads_started":1}
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:55.656719 17463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.651945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:42863)
0925 14:58:55.652138 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.652152 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:55.652231 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:55.652386 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:55.652387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:55.652583 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:55.652638 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.652640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.653381 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.653382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.653924 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.653926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.654292 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.654293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.654352 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.654800 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:55.654804 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:55.656626 (+ 1822us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":91,"threads_started":1}
W20260925 14:58:55.656798 17462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.652079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:42863 (local address 127.8.13.254:44567)
0925 14:58:55.652142 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.652144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.652217 (+ 73us) server_negotiation.cc:407] Connection header received
0925 14:58:55.652305 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.652306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.652319 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.652371 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.652711 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.653305 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.654006 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.654215 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.654274 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.656745 (+ 2471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:42863: BlockingRecv error: recv got EOF from 127.8.13.194:42863 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:55 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.195@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:55.774431 17464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:55.769909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.195:39955)
0925 14:58:55.770067 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:55.770079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:55.770154 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:55.770313 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:55.770314 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:55.770490 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:55.770536 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.770537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.771212 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.771213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.771786 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:55.771788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.772110 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.772111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:55.772163 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.772686 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:55.772691 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:55.774340 (+ 1649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
W20260925 14:58:55.774487 17462 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:55.770010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:39955 (local address 127.8.13.254:44567)
0925 14:58:55.770078 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:55.770080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:55.770154 (+ 74us) server_negotiation.cc:407] Connection header received
0925 14:58:55.770234 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:55.770235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:55.770251 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:55.770296 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:55.770611 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.771129 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.771866 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:55.772048 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:55.772109 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:55.774434 (+ 2325us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.195:39955: BlockingRecv error: recv got EOF from 127.8.13.195:39955 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 25 14:58:56 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:56 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.193@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:56.532145 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.524532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.193:51115)
0925 14:58:56.524702 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:56.524716 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:56.524782 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:56.524918 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:56.524918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:56.525097 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:56.525147 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:56.525148 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.525760 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.525761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:56.526295 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:56.526297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.526583 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.526583 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:56.526618 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:56.530190 (+ 3572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:56.530196 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:56.532080 (+ 1884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
W20260925 14:58:56.532274 17466 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:56.524630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:51115 (local address 127.8.13.254:44567)
0925 14:58:56.524773 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:56.524775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:56.524784 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:56.524828 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:56.524828 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:56.524846 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:56.524907 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:56.525202 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.525706 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.526351 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.526535 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.526578 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:56.532192 (+ 5614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.193:51115: BlockingRecv error: recv got EOF from 127.8.13.193:51115 (error 108)
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
Sep 25 14:58:56 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
Sep 25 14:58:56 dist-test-slave-15hz krb5kdc[16408](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.13.194@KRBTEST.COM for oryx/127.8.13.254@KRBTEST.COM, Server not found in Kerberos database
I20260925 14:58:56.661746 17467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.657298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44567 (local address 127.8.13.194:53707)
0925 14:58:56.657451 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:56.657460 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:56.657506 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:56.657627 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:56.657628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:56.657778 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:56.657828 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:56.657830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.658474 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.658476 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:56.659020 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:56.659022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.659465 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.659465 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:56.659504 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:56.660008 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:56.660012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:56.661684 (+ 1672us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.13.254:44567: Server oryx/127.8.13.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":63,"threads_started":1}
W20260925 14:58:56.661837 17466 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:56.657417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:53707 (local address 127.8.13.254:44567)
0925 14:58:56.657490 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:56.657492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:56.657501 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:56.657552 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:56.657553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:56.657564 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:56.657613 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:56.657877 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.658381 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.659187 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.659388 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.659462 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:56.661778 (+ 2316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.13.194:53707: BlockingRecv error: recv got EOF from 127.8.13.194:53707 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 14:58:56.706737 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16991
I20260925 14:58:56.710765 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17143
I20260925 14:58:56.714320 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17275
I20260925 14:58:56.717928 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 16921
2026-09-25T14:58:56Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17331 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-25T14:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:56Z Disabled control of system clock
I20260925 14:58:56.743245 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44763
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44763
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260925 14:58:56.821707 17471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:56.821939 17471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:56.821960 17471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:56.823168 17471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:56.823213 17471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:56.823225 17471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:56.823236 17471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:56.823247 17471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:56.824637 17471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:56.824890 17471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:56.825152 17471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:56.830335 17471 server_base.cc:1034] running on GCE node
W20260925 14:58:56.830335 17477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:56.830356 17476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:56.830519 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:56.830842 17471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:56.831162 17471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:56.832338 17471 hybrid_clock.cc:648] HybridClock initialized: now 1790348336832316 us; error 35 us; skew 500 ppm
I20260925 14:58:56.833709 17471 webserver.cc:460] Webserver started at http://127.8.13.254:34085/ using document root <none> and password file <none>
I20260925 14:58:56.833951 17471 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:56.833999 17471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:56.834151 17471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:56.835242 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "cd545766341c4520830badf535d3ee57"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "18631c61561717e2138e14db321a3eba"
server_key_iv: "3d793f6479ff73bc6ec9519c802d5791"
server_key_version: "encryptionkey@0"
I20260925 14:58:56.835615 17471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "cd545766341c4520830badf535d3ee57"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "18631c61561717e2138e14db321a3eba"
server_key_iv: "3d793f6479ff73bc6ec9519c802d5791"
server_key_version: "encryptionkey@0"
I20260925 14:58:56.836927 17471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:56.837718 17485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:56.837854 17471 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:56.837942 17471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "cd545766341c4520830badf535d3ee57"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "18631c61561717e2138e14db321a3eba"
server_key_iv: "3d793f6479ff73bc6ec9519c802d5791"
server_key_version: "encryptionkey@0"
I20260925 14:58:56.838016 17471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:56.855983 17471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:56.856331 17471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:56.856468 17471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:56.860152 17471 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44763
I20260925 14:58:56.860190 17537 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44763 every 8 connection(s)
I20260925 14:58:56.860510 17471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:56.861163 17538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:56.863768 17538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: Bootstrap starting.
I20260925 14:58:56.864344 17538 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:56.864609 17538 log.cc:824] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:56.865300 17538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: No bootstrap required, opened a new log
I20260925 14:58:56.866894 17538 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } }
I20260925 14:58:56.867081 17538 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:56.867105 17538 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd545766341c4520830badf535d3ee57, State: Initialized, Role: FOLLOWER
I20260925 14:58:56.867254 17538 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } }
I20260925 14:58:56.867324 17538 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:56.867348 17538 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:56.867395 17538 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:56.867594 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17471
I20260925 14:58:56.867687 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:56.867864 8247 external_mini_cluster.cc:1428] Setting key 3249364b7c3d3dc839a43ef118301490
I20260925 14:58:56.868067 17538 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } }
I20260925 14:58:56.868181 17538 leader_election.cc:302] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd545766341c4520830badf535d3ee57; no voters:
I20260925 14:58:56.868368 17538 leader_election.cc:288] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:56.868402 17541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:56.868602 17541 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [term 1 LEADER]: Becoming Leader. State: Replica: cd545766341c4520830badf535d3ee57, State: Running, Role: LEADER
I20260925 14:58:56.868690 17541 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } }
I20260925 14:58:56.869088 17541 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd545766341c4520830badf535d3ee57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } } }
I20260925 14:58:56.869155 17541 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:56.869308 17544 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd545766341c4520830badf535d3ee57. Latest consensus state: current_term: 1 leader_uuid: "cd545766341c4520830badf535d3ee57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd545766341c4520830badf535d3ee57" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44763 } } }
I20260925 14:58:56.869426 17544 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:56.869514 17538 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57 [sys.catalog]: configured and running, proceeding with master startup.
W20260925 14:58:56.871088 17558 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 14:58:56.871142 17558 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 14:58:56.871193 17546 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:56.871362 17546 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:56.873407 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:56.868378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56944 (local address 127.8.13.254:44763)
0925 14:58:56.868551 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:56.868562 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:56.868591 (+ 29us) server_negotiation.cc:407] Connection header received
0925 14:58:56.868724 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:56.868731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:56.868802 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:56.868951 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:56.869140 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.870257 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.871264 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:56.871898 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:56.871991 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:56.872820 (+ 829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:56.872823 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:56.872823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:56.872991 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:56.873025 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:56.873036 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:56.873091 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:56.873183 (+ 92us) server_negotiation.cc:299] Negotiation successful
0925 14:58:56.873247 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":77,"threads_started":1}
I20260925 14:58:56.875965 17546 catalog_manager.cc:1347] Generated new cluster ID: 8e594f15c2364ed3a9cb032787d5999a
I20260925 14:58:56.876029 17546 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:56.886931 17546 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:56.887579 17546 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:56.895170 17546 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd545766341c4520830badf535d3ee57: Generated new TSK 0
I20260925 14:58:56.895362 17546 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:56.904337 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--builtin_ntp_servers=127.8.13.212:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:58:56.984565 17561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:56.984804 17561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:56.984835 17561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:56.986008 17561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:56.986047 17561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:56.986079 17561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:56.987404 17561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:56.987672 17561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:56.987957 17561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:56.993455 17567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:56.993475 17566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:56.993434 17569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:56.993669 17561 server_base.cc:1034] running on GCE node
I20260925 14:58:56.993956 17561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:56.994217 17561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:56.995386 17561 hybrid_clock.cc:648] HybridClock initialized: now 1790348336995368 us; error 33 us; skew 500 ppm
I20260925 14:58:56.996790 17561 webserver.cc:460] Webserver started at http://127.8.13.193:46839/ using document root <none> and password file <none>
I20260925 14:58:56.997035 17561 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:56.997092 17561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:56.997213 17561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:56.998579 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "9cd2dcca22d8419290f72a171abe4335"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "3567912f17d7902742a3194a29bdc31f"
server_key_iv: "e8fbe62a34e65ead3249fc52b9f2cb5e"
server_key_version: "encryptionkey@0"
I20260925 14:58:56.998934 17561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "9cd2dcca22d8419290f72a171abe4335"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "3567912f17d7902742a3194a29bdc31f"
server_key_iv: "e8fbe62a34e65ead3249fc52b9f2cb5e"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.000262 17561 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:57.001008 17575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.001197 17561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 14:58:57.001279 17561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "9cd2dcca22d8419290f72a171abe4335"
format_stamp: "Formatted at 2026-09-25 14:58:56 on dist-test-slave-15hz"
server_key: "3567912f17d7902742a3194a29bdc31f"
server_key_iv: "e8fbe62a34e65ead3249fc52b9f2cb5e"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.001332 17561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.035785 17561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.036103 17561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.036213 17561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.036427 17561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.036748 17561 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.036783 17561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.036811 17561 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.036831 17561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.042407 17561 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:36389
I20260925 14:58:57.042496 17688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:36389 every 8 connection(s)
I20260925 14:58:57.042795 17561 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:57.047978 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.043601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:48781 (local address 127.8.13.254:44763)
0925 14:58:57.043661 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.043663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.043882 (+ 219us) server_negotiation.cc:407] Connection header received
0925 14:58:57.044394 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.044395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.044408 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.044455 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.045145 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.045663 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.046657 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.046864 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.046921 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.047751 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.047752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.047752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.047756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.047771 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.047775 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.047817 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.047882 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.047912 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:58:57.048173 17691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.043515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44763 (local address 127.8.13.193:48781)
0925 14:58:57.043757 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.043792 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.044312 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.044516 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.044520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.044529 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.045035 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.045040 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.045750 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.045751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.046567 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.046569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.046935 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.046935 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.047047 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.047619 (+ 572us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.047620 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.047650 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.047676 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.047823 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.047826 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.047828 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.047876 (+ 48us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.047960 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260925 14:58:57.048640 17689 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44763
I20260925 14:58:57.048727 17689 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.048893 17689 heartbeater.cc:511] Master 127.8.13.254:44763 requested a full tablet report, sending...
I20260925 14:58:57.049351 17502 ts_manager.cc:194] Registered new tserver with Master: 9cd2dcca22d8419290f72a171abe4335 (127.8.13.193:36389)
I20260925 14:58:57.049714 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17561
I20260925 14:58:57.049789 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:57.049971 8247 external_mini_cluster.cc:1428] Setting key 1f4dbb053dfdba0d688933600397e935
I20260925 14:58:57.050052 17502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.193:48781
I20260925 14:58:57.050804 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--builtin_ntp_servers=127.8.13.212:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:58:57.130761 17692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.130990 17692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.131017 17692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.132210 17692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.132253 17692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.132277 17692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:57.133577 17692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.133776 17692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.134044 17692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.138927 17698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.138955 17700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.138978 17697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.139213 17692 server_base.cc:1034] running on GCE node
I20260925 14:58:57.139415 17692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.139686 17692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.140822 17692 hybrid_clock.cc:648] HybridClock initialized: now 1790348337140808 us; error 27 us; skew 500 ppm
I20260925 14:58:57.142215 17692 webserver.cc:460] Webserver started at http://127.8.13.194:44765/ using document root <none> and password file <none>
I20260925 14:58:57.142455 17692 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.142509 17692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.142648 17692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.143805 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "3668581260a044ed951c2904dbcd9124"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "d055f9e83393c5728823e7be517424dd"
server_key_iv: "6b22e026ca6ba35bfa2ecf923d75bb9b"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.144152 17692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "3668581260a044ed951c2904dbcd9124"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "d055f9e83393c5728823e7be517424dd"
server_key_iv: "6b22e026ca6ba35bfa2ecf923d75bb9b"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.145421 17692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:57.146216 17706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.146378 17692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:57.146468 17692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "3668581260a044ed951c2904dbcd9124"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "d055f9e83393c5728823e7be517424dd"
server_key_iv: "6b22e026ca6ba35bfa2ecf923d75bb9b"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.146554 17692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.158188 17692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.158598 17692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.158752 17692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.159008 17692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.159348 17692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.159382 17692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.159405 17692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.159430 17692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.164819 17692 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:34023
I20260925 14:58:57.164868 17819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:34023 every 8 connection(s)
I20260925 14:58:57.165199 17692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:57.170198 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.165949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:47539 (local address 127.8.13.254:44763)
0925 14:58:57.166033 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.166035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.166238 (+ 203us) server_negotiation.cc:407] Connection header received
0925 14:58:57.166724 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.166725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.166734 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.166783 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.167424 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.168001 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.168942 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.169141 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.169194 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.169957 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.169958 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.169958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.169963 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.169980 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.169985 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.170049 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.170112 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.170137 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 14:58:57.170383 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.165916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44763 (local address 127.8.13.194:47539)
0925 14:58:57.166120 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.166156 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.166646 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.166838 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.166841 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.166850 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.167336 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.167341 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.168078 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.168079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.168873 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.168875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.169205 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.169205 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.169311 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.169815 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.169815 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.169844 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.169867 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.170035 (+ 168us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.170039 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.170042 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.170098 (+ 56us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.170182 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260925 14:58:57.170827 17820 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44763
I20260925 14:58:57.170915 17820 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.171105 17820 heartbeater.cc:511] Master 127.8.13.254:44763 requested a full tablet report, sending...
I20260925 14:58:57.171491 17502 ts_manager.cc:194] Registered new tserver with Master: 3668581260a044ed951c2904dbcd9124 (127.8.13.194:34023)
I20260925 14:58:57.171864 17502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.194:47539
I20260925 14:58:57.175365 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17692
I20260925 14:58:57.175442 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:57.175659 8247 external_mini_cluster.cc:1428] Setting key fa7fd3c219b9ef58a209cd947b5e0ef7
I20260925 14:58:57.176523 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--builtin_ntp_servers=127.8.13.212:39249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260925 14:58:57.256039 17823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.256260 17823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.256281 17823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.257472 17823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.257519 17823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.257542 17823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:57.258912 17823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44763
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.259158 17823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.259452 17823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.264207 17829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.264221 17831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.264230 17828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.264478 17823 server_base.cc:1034] running on GCE node
I20260925 14:58:57.264700 17823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.264920 17823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.266077 17823 hybrid_clock.cc:648] HybridClock initialized: now 1790348337266060 us; error 26 us; skew 500 ppm
I20260925 14:58:57.267388 17823 webserver.cc:460] Webserver started at http://127.8.13.195:35785/ using document root <none> and password file <none>
I20260925 14:58:57.267676 17823 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.267728 17823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.267859 17823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.269012 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "e64a2d78e6124cd19e78f2ee33aeac0c"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "c177f8a9fdb108ff7a7dd611318ea030"
server_key_iv: "e921dee1fdc5e1dc8dbb8c5231b42e41"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.269362 17823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "e64a2d78e6124cd19e78f2ee33aeac0c"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "c177f8a9fdb108ff7a7dd611318ea030"
server_key_iv: "e921dee1fdc5e1dc8dbb8c5231b42e41"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.270713 17823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:57.271412 17837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.271593 17823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:57.271682 17823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "e64a2d78e6124cd19e78f2ee33aeac0c"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "c177f8a9fdb108ff7a7dd611318ea030"
server_key_iv: "e921dee1fdc5e1dc8dbb8c5231b42e41"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.271751 17823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.292412 17823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.292734 17823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.292869 17823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.293076 17823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.293399 17823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.293432 17823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.293462 17823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.293495 17823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.298872 17823 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:46551
I20260925 14:58:57.298926 17950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:46551 every 8 connection(s)
I20260925 14:58:57.299268 17823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:57.301167 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17823
I20260925 14:58:57.301255 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:57.301414 8247 external_mini_cluster.cc:1428] Setting key eb5dd283d79b22d55057fc3b1ba48a1a
I20260925 14:58:57.304606 17543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.299981 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:60739 (local address 127.8.13.254:44763)
0925 14:58:57.300032 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.300034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.300273 (+ 239us) server_negotiation.cc:407] Connection header received
0925 14:58:57.300758 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.300759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.300768 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.300827 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.301579 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.302134 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.303199 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.303395 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.303449 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.304340 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.304341 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.304342 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.304346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.304362 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.304367 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.304409 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.304482 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.304511 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 14:58:57.304836 17953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.299963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44763 (local address 127.8.13.195:60739)
0925 14:58:57.300163 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.300202 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.300684 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.300883 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.300886 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.300894 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.301427 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.301433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.302210 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.302211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.303112 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:57.303114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:57.303459 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:57.303460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:57.303618 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:57.304204 (+ 586us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.304205 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.304242 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.304265 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.304416 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.304420 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.304423 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.304486 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.304594 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":73,"threads_started":1}
I20260925 14:58:57.305270 17951 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44763
I20260925 14:58:57.305356 17951 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.305531 17951 heartbeater.cc:511] Master 127.8.13.254:44763 requested a full tablet report, sending...
I20260925 14:58:57.305874 17502 ts_manager.cc:194] Registered new tserver with Master: e64a2d78e6124cd19e78f2ee33aeac0c (127.8.13.195:46551)
I20260925 14:58:57.306262 17502 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.195:60739
I20260925 14:58:57.313079 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:58:57.314808 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:57.314397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56954 (local address 127.8.13.254:44763)
0925 14:58:57.314470 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.314472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.314507 (+ 35us) server_negotiation.cc:407] Connection header received
0925 14:58:57.314563 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.314564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.314576 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.314633 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.314741 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56954: BlockingRecv error: recv got EOF from 127.0.0.1:56954 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 14:58:57.314877 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17561
I20260925 14:58:57.319394 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17692
I20260925 14:58:57.323125 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17823
I20260925 14:58:57.326846 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17471
2026-09-25T14:58:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (608 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-25T14:58:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:57Z Disabled control of system clock
I20260925 14:58:57.351938 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:36717
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:36717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:58:57.432163 17962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.432389 17962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.432411 17962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.433547 17962 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:57.433586 17962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.433598 17962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.433609 17962 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:57.433619 17962 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:57.434944 17962 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:36717
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:36717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.435165 17962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.435457 17962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.440013 17970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.439999 17967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.440192 17962 server_base.cc:1034] running on GCE node
W20260925 14:58:57.440233 17968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.440469 17962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.440727 17962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.441861 17962 hybrid_clock.cc:648] HybridClock initialized: now 1790348337441850 us; error 28 us; skew 500 ppm
I20260925 14:58:57.443075 17962 webserver.cc:460] Webserver started at http://127.8.13.254:36863/ using document root <none> and password file <none>
I20260925 14:58:57.443287 17962 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.443331 17962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.443456 17962 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.444530 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "cb09186451de4abb87e0034d56a85d02"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "f437a3e8a9dd5b6a5e3545ce38605fa5"
server_key_iv: "fbb095a82c8feb1cf40fe88324c8b7e2"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.444875 17962 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "cb09186451de4abb87e0034d56a85d02"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "f437a3e8a9dd5b6a5e3545ce38605fa5"
server_key_iv: "fbb095a82c8feb1cf40fe88324c8b7e2"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.446089 17962 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:57.446727 17976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.446882 17962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:57.446961 17962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "cb09186451de4abb87e0034d56a85d02"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "f437a3e8a9dd5b6a5e3545ce38605fa5"
server_key_iv: "fbb095a82c8feb1cf40fe88324c8b7e2"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.447032 17962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.450314 17962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.450572 17962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.450691 17962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.454393 17962 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:36717
I20260925 14:58:57.454425 18028 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:36717 every 8 connection(s)
I20260925 14:58:57.454744 17962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:57.455368 18029 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:57.455693 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 17962
I20260925 14:58:57.455771 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:57.455981 8247 external_mini_cluster.cc:1428] Setting key de1d89c283f77140741f6fe4124a758f
I20260925 14:58:57.458571 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.456537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34142 (local address 127.8.13.254:36717)
0925 14:58:57.456742 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.456751 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.456783 (+ 32us) server_negotiation.cc:407] Connection header received
0925 14:58:57.457251 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.457261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.457320 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.457483 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.458132 (+ 649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.458137 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.458137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.458142 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.458187 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.458201 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.458279 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.458315 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.458372 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260925 14:58:57.459053 18029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02: Bootstrap starting.
I20260925 14:58:57.459666 18029 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:57.459887 18029 log.cc:824] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:57.460471 18029 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02: No bootstrap required, opened a new log
I20260925 14:58:57.461833 18029 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } }
I20260925 14:58:57.461966 18029 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:57.461995 18029 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb09186451de4abb87e0034d56a85d02, State: Initialized, Role: FOLLOWER
I20260925 14:58:57.462119 18029 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } }
I20260925 14:58:57.462200 18029 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:57.462232 18029 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:57.462280 18029 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:57.462759 18029 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } }
I20260925 14:58:57.462850 18029 leader_election.cc:302] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb09186451de4abb87e0034d56a85d02; no voters:
I20260925 14:58:57.462994 18029 leader_election.cc:288] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:57.463259 18029 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:57.463256 18034 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:57.463420 18034 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [term 1 LEADER]: Becoming Leader. State: Replica: cb09186451de4abb87e0034d56a85d02, State: Running, Role: LEADER
I20260925 14:58:57.463521 18034 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } }
I20260925 14:58:57.463889 18036 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb09186451de4abb87e0034d56a85d02. Latest consensus state: current_term: 1 leader_uuid: "cb09186451de4abb87e0034d56a85d02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } } }
I20260925 14:58:57.463878 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb09186451de4abb87e0034d56a85d02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb09186451de4abb87e0034d56a85d02" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 36717 } } }
I20260925 14:58:57.463981 18036 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:57.463985 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:57.464171 18043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:58:57.464635 18043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:57.466058 18043 catalog_manager.cc:1347] Generated new cluster ID: 3e5861b129214c5cb206200a8f43be5a
I20260925 14:58:57.466100 18043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:57.482975 18043 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:57.483182 18043 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:57.490855 18043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb09186451de4abb87e0034d56a85d02: Generated new TSK 0
I20260925 14:58:57.491040 18043 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 14:58:57.493860 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--builtin_ntp_servers=127.8.13.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:58:57.583678 18053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.583895 18053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.583926 18053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.585156 18053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.585198 18053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.585233 18053 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:57.586597 18053 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.586839 18053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.587131 18053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.591938 18058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.591928 18059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.591930 18061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.592414 18053 server_base.cc:1034] running on GCE node
I20260925 14:58:57.592644 18053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.592873 18053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.594010 18053 hybrid_clock.cc:648] HybridClock initialized: now 1790348337593997 us; error 30 us; skew 500 ppm
I20260925 14:58:57.595207 18053 webserver.cc:460] Webserver started at http://127.8.13.193:41829/ using document root <none> and password file <none>
I20260925 14:58:57.595418 18053 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.595468 18053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.595635 18053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.596678 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "a088047ce8074bff99b98c351d331112"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "1060c855020f5caab6c7b50ba0242317"
server_key_iv: "f89017b49f03b12d81aab15f6b3bf8e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.597020 18053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "a088047ce8074bff99b98c351d331112"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "1060c855020f5caab6c7b50ba0242317"
server_key_iv: "f89017b49f03b12d81aab15f6b3bf8e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.598291 18053 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:57.599056 18067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.599216 18053 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:57.599298 18053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "a088047ce8074bff99b98c351d331112"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "1060c855020f5caab6c7b50ba0242317"
server_key_iv: "f89017b49f03b12d81aab15f6b3bf8e6"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.599364 18053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.602566 18053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.602819 18053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.602936 18053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.603124 18053 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.603403 18053 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.603434 18053 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.603461 18053 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.603484 18053 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.608986 18053 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:35541
I20260925 14:58:57.609071 18180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:35541 every 8 connection(s)
I20260925 14:58:57.609370 18053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:57.612238 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.610069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:37939 (local address 127.8.13.254:36717)
0925 14:58:57.610118 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.610120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.610366 (+ 246us) server_negotiation.cc:407] Connection header received
0925 14:58:57.610845 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.610846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.610852 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.610906 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.612063 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.612064 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.612064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.612067 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.612078 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.612081 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.612121 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.612170 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.612182 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:57.612419 18183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.610071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:36717 (local address 127.8.13.193:37939)
0925 14:58:57.610249 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.610284 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.610751 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.610973 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.610977 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.610985 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.611929 (+ 944us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.611930 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.611962 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.611993 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.612120 (+ 127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.612123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.612125 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.612174 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.612225 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
I20260925 14:58:57.612857 18181 heartbeater.cc:347] Connected to a master server at 127.8.13.254:36717
I20260925 14:58:57.612944 18181 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.613065 18181 heartbeater.cc:511] Master 127.8.13.254:36717 requested a full tablet report, sending...
I20260925 14:58:57.613543 17993 ts_manager.cc:194] Registered new tserver with Master: a088047ce8074bff99b98c351d331112 (127.8.13.193:35541)
I20260925 14:58:57.619419 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18053
I20260925 14:58:57.619519 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:57.619763 8247 external_mini_cluster.cc:1428] Setting key 3a4ae27f282576809ced9f218a0e093d
I20260925 14:58:57.620721 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--builtin_ntp_servers=127.8.13.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:58:57.700860 18184 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.701092 18184 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.701116 18184 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.702459 18184 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.702513 18184 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.702541 18184 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:57.704090 18184 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18184
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.704340 18184 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.704694 18184 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.709523 18190 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.709517 18192 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.709518 18189 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.709594 18184 server_base.cc:1034] running on GCE node
I20260925 14:58:57.709931 18184 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.710170 18184 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.711305 18184 hybrid_clock.cc:648] HybridClock initialized: now 1790348337711291 us; error 31 us; skew 500 ppm
I20260925 14:58:57.712561 18184 webserver.cc:460] Webserver started at http://127.8.13.194:34647/ using document root <none> and password file <none>
I20260925 14:58:57.712779 18184 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.712826 18184 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.712953 18184 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.714012 18184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "5efd9f5ffc6747e9aca387607e0b9073"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "7fa683a42cc93721f10fa3e4e5fe040c"
server_key_iv: "bb5d8f64e7764b6de6ff0ad8552539f0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.714349 18184 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "5efd9f5ffc6747e9aca387607e0b9073"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "7fa683a42cc93721f10fa3e4e5fe040c"
server_key_iv: "bb5d8f64e7764b6de6ff0ad8552539f0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.715637 18184 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:58:57.716297 18198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.716434 18184 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:57.716516 18184 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "5efd9f5ffc6747e9aca387607e0b9073"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "7fa683a42cc93721f10fa3e4e5fe040c"
server_key_iv: "bb5d8f64e7764b6de6ff0ad8552539f0"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.716580 18184 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.719918 18184 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.720173 18184 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.720290 18184 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.720481 18184 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.720764 18184 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.720798 18184 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.720824 18184 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.720847 18184 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.726154 18184 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:35379
I20260925 14:58:57.726199 18311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:35379 every 8 connection(s)
I20260925 14:58:57.726523 18184 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:58:57.729579 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.727236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:44535 (local address 127.8.13.254:36717)
0925 14:58:57.727285 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.727287 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.727522 (+ 235us) server_negotiation.cc:407] Connection header received
0925 14:58:57.728045 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.728047 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.728054 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.728106 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.729330 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.729332 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.729332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.729338 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.729356 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.729360 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.729410 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.729491 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.729506 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 14:58:57.729753 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.727218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:36717 (local address 127.8.13.194:44535)
0925 14:58:57.727400 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.727444 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.727935 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.728176 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.728180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.728188 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.729165 (+ 977us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.729165 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.729205 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.729237 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.729416 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.729421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.729424 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.729492 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.729555 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260925 14:58:57.730293 18312 heartbeater.cc:347] Connected to a master server at 127.8.13.254:36717
I20260925 14:58:57.730383 18312 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.730530 18312 heartbeater.cc:511] Master 127.8.13.254:36717 requested a full tablet report, sending...
I20260925 14:58:57.730906 17993 ts_manager.cc:194] Registered new tserver with Master: 5efd9f5ffc6747e9aca387607e0b9073 (127.8.13.194:35379)
I20260925 14:58:57.735455 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18184
I20260925 14:58:57.735589 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:57.735780 8247 external_mini_cluster.cc:1428] Setting key 558ca98e06e31d0bdb2589cecfd42e26
I20260925 14:58:57.736780 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--builtin_ntp_servers=127.8.13.212:39191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260925 14:58:57.815171 18315 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:57.815389 18315 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:57.815420 18315 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:57.816649 18315 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:57.816690 18315 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:57.816725 18315 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:57.818075 18315 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:36717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18315
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:57.818311 18315 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:57.818596 18315 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:57.823611 18321 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.823611 18323 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:57.823590 18320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:57.823756 18315 server_base.cc:1034] running on GCE node
I20260925 14:58:57.824036 18315 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:57.824275 18315 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:57.825419 18315 hybrid_clock.cc:648] HybridClock initialized: now 1790348337825406 us; error 29 us; skew 500 ppm
I20260925 14:58:57.826674 18315 webserver.cc:460] Webserver started at http://127.8.13.195:43107/ using document root <none> and password file <none>
I20260925 14:58:57.826895 18315 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:57.826944 18315 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:57.827073 18315 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:57.828275 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "87a379ac7b5d46f7997aca0650cfb915"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "52ae58825ed64f32d7fd925cba9e67ad"
server_key_iv: "5e96130c61b6f81211c285220e661dbf"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.828678 18315 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "87a379ac7b5d46f7997aca0650cfb915"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "52ae58825ed64f32d7fd925cba9e67ad"
server_key_iv: "5e96130c61b6f81211c285220e661dbf"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.830237 18315 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:57.831030 18329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.831220 18315 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:58:57.831322 18315 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "87a379ac7b5d46f7997aca0650cfb915"
format_stamp: "Formatted at 2026-09-25 14:58:57 on dist-test-slave-15hz"
server_key: "52ae58825ed64f32d7fd925cba9e67ad"
server_key_iv: "5e96130c61b6f81211c285220e661dbf"
server_key_version: "encryptionkey@0"
I20260925 14:58:57.831393 18315 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:57.836048 18315 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:57.836341 18315 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:57.836483 18315 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:57.836669 18315 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:57.836943 18315 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:57.836985 18315 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.837011 18315 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:57.837033 18315 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:57.842293 18315 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:35951
I20260925 14:58:57.842356 18442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:35951 every 8 connection(s)
I20260925 14:58:57.842640 18315 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:58:57.845345 18032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.843280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:58471 (local address 127.8.13.254:36717)
0925 14:58:57.843325 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.843327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.843593 (+ 266us) server_negotiation.cc:407] Connection header received
0925 14:58:57.844054 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.844055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.844061 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.844120 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.845169 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:57.845170 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:57.845171 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 14:58:57.845174 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:57.845189 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 14:58:57.845193 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:57.845237 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:57.845280 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 14:58:57.845291 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:58:57.845525 18445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:57.843268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:36717 (local address 127.8.13.195:58471)
0925 14:58:57.843440 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:57.843490 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:57.843974 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:57.844174 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:57.844177 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:57.844185 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:57.845035 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 14:58:57.845035 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:57.845063 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 14:58:57.845092 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:57.845236 (+ 144us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:57.845238 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:57.845240 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:57.845287 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 14:58:57.845333 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260925 14:58:57.845968 18443 heartbeater.cc:347] Connected to a master server at 127.8.13.254:36717
I20260925 14:58:57.846050 18443 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:57.846164 18443 heartbeater.cc:511] Master 127.8.13.254:36717 requested a full tablet report, sending...
I20260925 14:58:57.846498 17993 ts_manager.cc:194] Registered new tserver with Master: 87a379ac7b5d46f7997aca0650cfb915 (127.8.13.195:35951)
I20260925 14:58:57.851056 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18315
I20260925 14:58:57.851132 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:57.851281 8247 external_mini_cluster.cc:1428] Setting key 788472a874fc6518fdd7b87690b44d87
I20260925 14:58:57.852147 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260925 14:58:57.853907 18032 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 14:58:57.853435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34150 (local address 127.8.13.254:36717)
0925 14:58:57.853490 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:57.853492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:57.853643 (+ 151us) server_negotiation.cc:407] Connection header received
0925 14:58:57.853688 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:57.853689 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:57.853694 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:57.853737 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:57.853845 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34150: BlockingRecv error: recv got EOF from 127.0.0.1:34150 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 14:58:57.853971 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18053
I20260925 14:58:57.858697 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18184
I20260925 14:58:57.862695 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18315
I20260925 14:58:57.866359 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 17962
2026-09-25T14:58:57Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (539 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:57 dist-test-slave-15hz krb5kdc[18453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:57 dist-test-slave-15hz krb5kdc[18453](info): set up 2 sockets
Sep 25 14:58:57 dist-test-slave-15hz krb5kdc[18453](info): commencing operation
krb5kdc: starting...
W20260925 14:58:58.812072 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:58:58 dist-test-slave-15hz krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:58:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:58:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:58:58.927857 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43175
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:43175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.007650 18466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.007866 18466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.007889 18466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.009100 18466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:58:59.009142 18466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.009155 18466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.009166 18466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:58:59.009176 18466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:58:59.010661 18466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:43175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:59.010893 18466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.011112 18466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:58:59.015795 18466 server_base.cc:1034] running on GCE node
W20260925 14:58:59.015763 18474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.015769 18472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.015810 18471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.016296 18466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.016552 18466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.017701 18466 hybrid_clock.cc:648] HybridClock initialized: now 1790348339017677 us; error 43 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.020169 18466 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:58:59.020604 18466 webserver.cc:460] Webserver started at http://127.8.13.254:39305/ using document root <none> and password file <none>
I20260925 14:58:59.020803 18466 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.020850 18466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.020972 18466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.021998 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "09e4a39b6a034e18976df18fae296661"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "1865075e85f7244b6b39063e711b64e5"
server_key_iv: "a04c69c4880a2825e466ff59427ad2a5"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.022324 18466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "09e4a39b6a034e18976df18fae296661"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "1865075e85f7244b6b39063e711b64e5"
server_key_iv: "a04c69c4880a2825e466ff59427ad2a5"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.023545 18466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 14:58:59.024307 18481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.024479 18466 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:59.024569 18466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "09e4a39b6a034e18976df18fae296661"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "1865075e85f7244b6b39063e711b64e5"
server_key_iv: "a04c69c4880a2825e466ff59427ad2a5"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.024632 18466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.042846 18466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.045122 18466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.045265 18466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.049070 18466 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:43175
I20260925 14:58:59.049118 18533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:43175 every 8 connection(s)
I20260925 14:58:59.049453 18466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:58:59.050455 18534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.052453 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18466
I20260925 14:58:59.052537 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:58:59.052709 8247 external_mini_cluster.cc:1428] Setting key 324f2d74afdd0e6141132c145b314ecf
I20260925 14:58:59.052850 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: Bootstrap starting.
I20260925 14:58:59.053576 18534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.053887 18534 log.cc:824] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.054576 18534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: No bootstrap required, opened a new log
I20260925 14:58:59.056396 18534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } }
I20260925 14:58:59.056535 18534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.056576 18534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e4a39b6a034e18976df18fae296661, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.056703 18534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } }
I20260925 14:58:59.056788 18534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:58:59.056829 18534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:58:59.056875 18534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.057403 18534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } }
I20260925 14:58:59.057509 18534 leader_election.cc:302] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e4a39b6a034e18976df18fae296661; no voters:
I20260925 14:58:59.057683 18534 leader_election.cc:288] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:58:59.057716 18539 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:59.057884 18539 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [term 1 LEADER]: Becoming Leader. State: Replica: 09e4a39b6a034e18976df18fae296661, State: Running, Role: LEADER
I20260925 14:58:59.057945 18534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:58:59.058012 18539 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } }
I20260925 14:58:59.058352 18539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e4a39b6a034e18976df18fae296661. Latest consensus state: current_term: 1 leader_uuid: "09e4a39b6a034e18976df18fae296661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } } }
I20260925 14:58:59.058581 18539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:59.058521 18540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e4a39b6a034e18976df18fae296661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e4a39b6a034e18976df18fae296661" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43175 } } }
I20260925 14:58:59.059031 18540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661 [sys.catalog]: This master's current role is: LEADER
I20260925 14:58:59.059186 18548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:59.059702 18548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:58:59.061589 18548 catalog_manager.cc:1347] Generated new cluster ID: 70b28d8a8dc6413f9ec784432d04f128
I20260925 14:58:59.061632 18548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:58:59.064181 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.053233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.8.13.254:43175)
0925 14:58:59.053410 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.053419 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.053455 (+ 36us) server_negotiation.cc:407] Connection header received
0925 14:58:59.054024 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.054033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.054101 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.054245 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.054875 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.055809 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.056552 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.057089 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.057191 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.060032 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.060037 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.060037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.060049 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.061993 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.062426 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.062429 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.062429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.062550 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.062780 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.062781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.062781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.063091 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.063725 (+ 634us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.063802 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.063960 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260925 14:58:59.068759 18548 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:58:59.069252 18548 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:58:59.073724 18548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: Generated new TSK 0
I20260925 14:58:59.073894 18548 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:58:59.103766 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.183732 18560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.183941 18560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.183972 18560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.185137 18560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.185174 18560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.185209 18560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:58:59.186619 18560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:59.186854 18560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.187057 18560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.191772 18568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.191820 18565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.191790 18566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.191926 18560 server_base.cc:1034] running on GCE node
I20260925 14:58:59.192193 18560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.192412 18560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.193544 18560 hybrid_clock.cc:648] HybridClock initialized: now 1790348339193534 us; error 27 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.195974 18560 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:58:59.196358 18560 webserver.cc:460] Webserver started at http://127.8.13.193:42935/ using document root <none> and password file <none>
I20260925 14:58:59.196549 18560 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.196588 18560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.196683 18560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.197666 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "d68df6c844ce045f0ca8c41d682678bc"
server_key_iv: "19f7543a0f3e4ba6b1e1f913e1ea2f23"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.197958 18560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "d68df6c844ce045f0ca8c41d682678bc"
server_key_iv: "19f7543a0f3e4ba6b1e1f913e1ea2f23"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.199191 18560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:59.199833 18575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.199980 18560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.200057 18560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "d68df6c844ce045f0ca8c41d682678bc"
server_key_iv: "19f7543a0f3e4ba6b1e1f913e1ea2f23"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.200110 18560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.225715 18560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.227954 18560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.228084 18560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.228304 18560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.228582 18560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.228612 18560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.228639 18560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.228655 18560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.233903 18560 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:43613
I20260925 14:58:59.233974 18688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:43613 every 8 connection(s)
I20260925 14:58:59.234275 18560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:58:59.239163 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18560
I20260925 14:58:59.239279 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:58:59.239567 8247 external_mini_cluster.cc:1428] Setting key fca7dce26ee42e752682ee37420c5296
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:59.243794 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.234997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:50213 (local address 127.8.13.254:43175)
0925 14:58:59.235065 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.235067 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.235236 (+ 169us) server_negotiation.cc:407] Connection header received
0925 14:58:59.235779 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.235780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.235790 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.235841 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.236774 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.237280 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.238262 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.238468 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.238539 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.241081 (+ 2542us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.241082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.241083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.241090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.242446 (+ 1356us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.242813 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.242814 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.242815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.242880 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.243147 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.243148 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.243148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.243275 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.243340 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.243605 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.243697 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 14:58:59.243907 18691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.234962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43175 (local address 127.8.13.193:50213)
0925 14:58:59.235114 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.235151 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.235681 (+ 530us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.235915 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.235918 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.236491 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.236694 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.236699 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.237352 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.237353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.238162 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.238164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.238575 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.238576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.238697 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.239207 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.239218 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.240941 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.242529 (+ 1588us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.242535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.242537 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.242748 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.242943 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.242944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.242944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.243092 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.243333 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.243336 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.243455 (+ 119us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.243569 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.243720 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260925 14:58:59.244410 18689 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43175
I20260925 14:58:59.244503 18689 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.244684 18689 heartbeater.cc:511] Master 127.8.13.254:43175 requested a full tablet report, sending...
I20260925 14:58:59.245226 18498 ts_manager.cc:194] Registered new tserver with Master: 8097fb4d9fcf4ec9b83b19f2cecfbb53 (127.8.13.193:43613)
I20260925 14:58:59.246176 18498 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:50213
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:58:59.266481 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.346448 18694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.346688 18694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.346717 18694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.347941 18694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.347990 18694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.348026 18694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:58:59.349495 18694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18694
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:59.349740 18694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.349959 18694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.354873 18699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.354887 18702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.355123 18694 server_base.cc:1034] running on GCE node
W20260925 14:58:59.355204 18700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.355402 18694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.355660 18694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.356807 18694 hybrid_clock.cc:648] HybridClock initialized: now 1790348339356796 us; error 29 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.359217 18694 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:58:59.359704 18694 webserver.cc:460] Webserver started at http://127.8.13.194:36465/ using document root <none> and password file <none>
I20260925 14:58:59.359900 18694 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.359941 18694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.360039 18694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.361058 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "e4997f0ce4764d0b91e9546c18ced9ba"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "c7540c6ce94d50809095f97026e78ca6"
server_key_iv: "aecf60448e0a9e5acb70350a20b4b126"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.361348 18694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "e4997f0ce4764d0b91e9546c18ced9ba"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "c7540c6ce94d50809095f97026e78ca6"
server_key_iv: "aecf60448e0a9e5acb70350a20b4b126"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.362701 18694 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 14:58:59.363468 18709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.363641 18694 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 14:58:59.363727 18694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "e4997f0ce4764d0b91e9546c18ced9ba"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "c7540c6ce94d50809095f97026e78ca6"
server_key_iv: "aecf60448e0a9e5acb70350a20b4b126"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.363796 18694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.373083 18694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.375077 18694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.375219 18694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.375434 18694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.375780 18694 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.375818 18694 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.375847 18694 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.375871 18694 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.381840 18694 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:43921
I20260925 14:58:59.381884 18822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:43921 every 8 connection(s)
I20260925 14:58:59.382205 18694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:59.390969 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.382984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:54877 (local address 127.8.13.254:43175)
0925 14:58:59.383028 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.383030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.383242 (+ 212us) server_negotiation.cc:407] Connection header received
0925 14:58:59.383808 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.383810 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.383821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.383869 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.384838 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.385333 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.386316 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.386506 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.386562 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.388666 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.388667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.388668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.388676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.389766 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.390112 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.390113 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.390113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.390164 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.390368 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.390369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.390370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.390478 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.390534 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.390790 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.390877 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:59.391144 18825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.382959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43175 (local address 127.8.13.194:54877)
0925 14:58:59.383123 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.383159 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.383716 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.383945 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.383950 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.384516 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.384739 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.384745 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.385415 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.385416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.386234 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.386236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.386583 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.386584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.386701 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.387168 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.387176 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.388582 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.389837 (+ 1255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.389843 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.389846 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.390046 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.390217 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.390218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.390218 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.390315 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.390531 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.390538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.390677 (+ 139us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.390765 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.390919 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260925 14:58:59.391350 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18694
I20260925 14:58:59.391456 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:58:59.391650 18823 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43175
I20260925 14:58:59.391683 8247 external_mini_cluster.cc:1428] Setting key ed7e2646c3677aaababfd35a0ccda68c
I20260925 14:58:59.391748 18823 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.391930 18823 heartbeater.cc:511] Master 127.8.13.254:43175 requested a full tablet report, sending...
I20260925 14:58:59.392300 18497 ts_manager.cc:194] Registered new tserver with Master: e4997f0ce4764d0b91e9546c18ced9ba (127.8.13.194:43921)
I20260925 14:58:59.392659 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:54877
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:58:59.417026 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:58:59.496003 18828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:58:59.496212 18828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:58:59.496232 18828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:58:59.497339 18828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:58:59.497373 18828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:58:59.497397 18828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:58:59.498817 18828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:43175
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:58:59.499011 18828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:58:59.499188 18828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:58:59.503923 18834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.504012 18833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:58:59.503925 18836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:58:59.504158 18828 server_base.cc:1034] running on GCE node
I20260925 14:58:59.504352 18828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:58:59.504590 18828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:58:59.505748 18828 hybrid_clock.cc:648] HybridClock initialized: now 1790348339505735 us; error 34 us; skew 500 ppm
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:58:59.508266 18828 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:58:59.508715 18828 webserver.cc:460] Webserver started at http://127.8.13.195:38945/ using document root <none> and password file <none>
I20260925 14:58:59.508927 18828 fs_manager.cc:359] Metadata directory not provided
I20260925 14:58:59.508980 18828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:58:59.509111 18828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 14:58:59.510138 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "7c395973ec204eea94b7fd8c4bc89c31"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "7a19358b9ef5fdf758a00a6f29e0925b"
server_key_iv: "9e0fa7d720de3f6c0972d8243cd131c6"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.510465 18828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "7c395973ec204eea94b7fd8c4bc89c31"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "7a19358b9ef5fdf758a00a6f29e0925b"
server_key_iv: "9e0fa7d720de3f6c0972d8243cd131c6"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.511729 18828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 14:58:59.512422 18843 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.512585 18828 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 14:58:59.512665 18828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "7c395973ec204eea94b7fd8c4bc89c31"
format_stamp: "Formatted at 2026-09-25 14:58:59 on dist-test-slave-15hz"
server_key: "7a19358b9ef5fdf758a00a6f29e0925b"
server_key_iv: "9e0fa7d720de3f6c0972d8243cd131c6"
server_key_version: "encryptionkey@0"
I20260925 14:58:59.512732 18828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:58:59.555553 18828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:58:59.557720 18828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:58:59.557866 18828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:58:59.558102 18828 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:58:59.558383 18828 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:58:59.558418 18828 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.558444 18828 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:58:59.558455 18828 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:58:59.564006 18828 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:42989
I20260925 14:58:59.564081 18956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:42989 every 8 connection(s)
I20260925 14:58:59.564373 18828 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[18453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348339, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:58:59.572297 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 18828
I20260925 14:58:59.572395 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:58:59.572589 8247 external_mini_cluster.cc:1428] Setting key 50331fa1b4dfd7dd728a204503cab871
I20260925 14:58:59.574057 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.565157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:34173 (local address 127.8.13.254:43175)
0925 14:58:59.565200 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.565202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.565445 (+ 243us) server_negotiation.cc:407] Connection header received
0925 14:58:59.566045 (+ 600us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.566046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.566059 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.566112 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.567131 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.567756 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.568784 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.568976 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.569041 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.571269 (+ 2228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.571270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.571271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.571280 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.572445 (+ 1165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.572953 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.572954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.572954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.573009 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.573247 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.573248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.573248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.573377 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.573441 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.573858 (+ 417us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.573960 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 14:58:59.574328 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.565146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:43175 (local address 127.8.13.195:34173)
0925 14:58:59.565319 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.565354 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.565911 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.566194 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.566198 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.566821 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:58:59.567029 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.567034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.567854 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.567855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.568697 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.568699 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.569051 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.569052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.569177 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.569719 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:58:59.569728 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:58:59.571169 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:58:59.572531 (+ 1362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.572540 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.572543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.572838 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.573078 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:58:59.573079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:58:59.573079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:58:59.573197 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:58:59.573539 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:58:59.573545 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:58:59.573731 (+ 186us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.573827 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.573995 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260925 14:58:59.574759 18957 heartbeater.cc:347] Connected to a master server at 127.8.13.254:43175
I20260925 14:58:59.574841 18957 heartbeater.cc:464] Registering TS with master...
I20260925 14:58:59.575006 18957 heartbeater.cc:511] Master 127.8.13.254:43175 requested a full tablet report, sending...
I20260925 14:58:59.575369 18497 ts_manager.cc:194] Registered new tserver with Master: 7c395973ec204eea94b7fd8c4bc89c31 (127.8.13.195:42989)
I20260925 14:58:59.575793 18497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:34173
I20260925 14:58:59.584496 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:58:59.591697 18537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.585786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.8.13.254:43175)
0925 14:58:59.585876 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.585878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.585969 (+ 91us) server_negotiation.cc:407] Connection header received
0925 14:58:59.586051 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.586052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.586062 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.586106 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:58:59.586645 (+ 539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.587150 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.587906 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.588109 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.588162 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.589346 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:58:59.589348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:58:59.589348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:58:59.589356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:58:59.590600 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.590907 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.590908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.590908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.590960 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:58:59.591123 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:58:59.591124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:58:59.591125 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:58:59.591274 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:58:59.591360 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.591492 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.591603 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:58:59.593309 18497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:58:59.594867 18497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:58:59.601248 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41580 (local address 127.8.13.193:43613)
0925 14:58:59.597645 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.597650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.597660 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:58:59.597696 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.597697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.597710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.597785 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.598027 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.598855 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.600236 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.600871 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.600933 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.600994 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.601127 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.601168 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260925 14:58:59.601239 18970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43613 (local address 127.0.0.1:41580)
0925 14:58:59.597387 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.597429 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.597493 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.597805 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.597808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.597811 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.597884 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.597886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.598943 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.598944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.600164 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.600166 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.600954 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.600955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.601030 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.601043 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.601107 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.601162 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":40,"threads_started":1}
I20260925 14:58:59.601573 18969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:42989 (local address 127.0.0.1:51012)
0925 14:58:59.597597 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.597606 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.597642 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.597890 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.597891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.597894 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.597947 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.597949 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.599286 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.599287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.600348 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.600350 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.601220 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.601222 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.601283 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.601287 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.601456 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.601492 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":5,"thread_start_us":94,"threads_started":1}
I20260925 14:58:59.601649 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46174 (local address 127.8.13.194:43921)
0925 14:58:59.597928 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.597932 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.597941 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:59.597988 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.597990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.598005 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.598143 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.598348 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.599382 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.600493 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.601193 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.601250 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.601307 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.601535 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.601578 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":66,"threads_started":1}
I20260925 14:58:59.601784 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:43921 (local address 127.0.0.1:46174)
0925 14:58:59.597640 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.597649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.597687 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.598224 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.598225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.598227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.598269 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.598271 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.599468 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.599469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.600425 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.600427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.601336 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.601337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.601395 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.601398 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.601443 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.601494 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":104,"threads_started":1}
I20260925 14:58:59.601893 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.597312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51012 (local address 127.8.13.195:42989)
0925 14:58:59.597756 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.597759 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.597766 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:58:59.597795 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.597804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.597819 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.597880 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.598027 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.599182 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.600416 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.601119 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.601684 (+ 565us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.601754 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.601778 (+ 24us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.601820 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":66,"threads_started":1}
I20260925 14:58:59.602052 18623 tablet_service.cc:1467] Processing CreateTablet for tablet 545a4fd4c03241af9e2a13f4034b324a (DEFAULT_TABLE table=test-table [id=01c8357b1b614498b674b467e42eff34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.602437 18623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545a4fd4c03241af9e2a13f4034b324a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.602439 18757 tablet_service.cc:1467] Processing CreateTablet for tablet 545a4fd4c03241af9e2a13f4034b324a (DEFAULT_TABLE table=test-table [id=01c8357b1b614498b674b467e42eff34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.602731 18757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545a4fd4c03241af9e2a13f4034b324a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.605301 18975 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Bootstrap starting.
I20260925 14:58:59.605720 18891 tablet_service.cc:1467] Processing CreateTablet for tablet 545a4fd4c03241af9e2a13f4034b324a (DEFAULT_TABLE table=test-table [id=01c8357b1b614498b674b467e42eff34]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:58:59.605885 18975 tablet_bootstrap.cc:654] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.605986 18891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 545a4fd4c03241af9e2a13f4034b324a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:58:59.606343 18975 log.cc:824] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.606711 18974 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Bootstrap starting.
I20260925 14:58:59.607231 18974 tablet_bootstrap.cc:654] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.607721 18974 log.cc:824] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.607901 18975 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: No bootstrap required, opened a new log
I20260925 14:58:59.607963 18975 ts_tablet_manager.cc:1397] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260925 14:58:59.608533 18978 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Bootstrap starting.
I20260925 14:58:59.608697 18974 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: No bootstrap required, opened a new log
I20260925 14:58:59.608768 18974 ts_tablet_manager.cc:1397] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:59.609200 18978 tablet_bootstrap.cc:654] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Neither blocks nor log segments found. Creating new log.
I20260925 14:58:59.609611 18978 log.cc:824] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Log is configured to *not* fsync() on all Append() calls
I20260925 14:58:59.609997 18975 raft_consensus.cc:348] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.610178 18978 tablet_bootstrap.cc:492] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: No bootstrap required, opened a new log
I20260925 14:58:59.610172 18975 raft_consensus.cc:374] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.610206 18975 raft_consensus.cc:729] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8097fb4d9fcf4ec9b83b19f2cecfbb53, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.610231 18978 ts_tablet_manager.cc:1397] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 14:58:59.610327 18975 consensus_queue.cc:260] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.610605 18689 heartbeater.cc:503] Master 127.8.13.254:43175 was elected leader, sending a full tablet report...
I20260925 14:58:59.610666 18974 raft_consensus.cc:348] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.610813 18974 raft_consensus.cc:374] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.610844 18974 raft_consensus.cc:729] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4997f0ce4764d0b91e9546c18ced9ba, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.610868 18975 ts_tablet_manager.cc:1428] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 14:58:59.610970 18974 consensus_queue.cc:260] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.611241 18823 heartbeater.cc:503] Master 127.8.13.254:43175 was elected leader, sending a full tablet report...
I20260925 14:58:59.611230 18974 ts_tablet_manager.cc:1428] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:59.612073 18978 raft_consensus.cc:348] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.612223 18978 raft_consensus.cc:374] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:58:59.612259 18978 raft_consensus.cc:729] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c395973ec204eea94b7fd8c4bc89c31, State: Initialized, Role: FOLLOWER
I20260925 14:58:59.612362 18978 consensus_queue.cc:260] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.612591 18978 ts_tablet_manager.cc:1428] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:58:59.612641 18957 heartbeater.cc:503] Master 127.8.13.254:43175 was elected leader, sending a full tablet report...
W20260925 14:58:59.632794 18824 tablet.cc:2369] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:58:59.735050 18690 tablet.cc:2369] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:58:59.753211 18982 raft_consensus.cc:482] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:58:59.753373 18982 raft_consensus.cc:504] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.753748 18982 leader_election.cc:288] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8097fb4d9fcf4ec9b83b19f2cecfbb53 (127.8.13.193:43613), e4997f0ce4764d0b91e9546c18ced9ba (127.8.13.194:43921)
I20260925 14:58:59.756600 18972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.754071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:53601 (local address 127.8.13.193:43613)
0925 14:58:59.754149 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.754151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.754160 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:58:59.754187 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.754188 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.754200 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.754253 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.754478 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.754946 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.755932 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.756292 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.756353 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.756362 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.756489 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.756526 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 14:58:59.756600 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.754004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43613 (local address 127.8.13.195:53601)
0925 14:58:59.754053 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.754065 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.754107 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.754294 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.754295 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.754298 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.754373 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.754375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.755028 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.755029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.755851 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.755853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.756360 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.756361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.756430 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.756432 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.756476 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.756521 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 14:58:59.756799 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.754004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:43921 (local address 127.8.13.195:52399)
0925 14:58:59.754352 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.754363 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:58:59.754433 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:58:59.754572 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:58:59.754573 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:58:59.754574 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:58:59.754631 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.754632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.755245 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.755246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.756031 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:58:59.756034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.756562 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.756563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:58:59.756622 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.756624 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 14:58:59.756673 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 14:58:59.756710 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"mutex_wait_us":147,"thread_start_us":111,"threads_started":1}
I20260925 14:58:59.756912 18973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.754082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:52399 (local address 127.8.13.194:43921)
0925 14:58:59.754127 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.754129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.754441 (+ 312us) server_negotiation.cc:407] Connection header received
0925 14:58:59.754504 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.754505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.754519 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.754561 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:58:59.754692 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.755167 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.756107 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.756485 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.756553 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.756561 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.756806 (+ 245us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.756849 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 14:58:59.756934 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545a4fd4c03241af9e2a13f4034b324a" candidate_uuid: "7c395973ec204eea94b7fd8c4bc89c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" is_pre_election: true
I20260925 14:58:59.757084 18643 raft_consensus.cc:2454] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c395973ec204eea94b7fd8c4bc89c31 in term 0.
I20260925 14:58:59.757169 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545a4fd4c03241af9e2a13f4034b324a" candidate_uuid: "7c395973ec204eea94b7fd8c4bc89c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" is_pre_election: true
I20260925 14:58:59.757344 18777 raft_consensus.cc:2454] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c395973ec204eea94b7fd8c4bc89c31 in term 0.
I20260925 14:58:59.757344 18845 leader_election.cc:302] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c395973ec204eea94b7fd8c4bc89c31, 8097fb4d9fcf4ec9b83b19f2cecfbb53; no voters:
I20260925 14:58:59.757459 18982 raft_consensus.cc:2781] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:58:59.757514 18982 raft_consensus.cc:482] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:58:59.757537 18982 raft_consensus.cc:3037] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.758265 18982 raft_consensus.cc:504] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.758390 18982 leader_election.cc:288] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [CANDIDATE]: Term 1 election: Requested vote from peers 8097fb4d9fcf4ec9b83b19f2cecfbb53 (127.8.13.193:43613), e4997f0ce4764d0b91e9546c18ced9ba (127.8.13.194:43921)
I20260925 14:58:59.758564 18777 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545a4fd4c03241af9e2a13f4034b324a" candidate_uuid: "7c395973ec204eea94b7fd8c4bc89c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4997f0ce4764d0b91e9546c18ced9ba"
I20260925 14:58:59.758644 18777 raft_consensus.cc:3037] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.758705 18643 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "545a4fd4c03241af9e2a13f4034b324a" candidate_uuid: "7c395973ec204eea94b7fd8c4bc89c31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53"
I20260925 14:58:59.758792 18643 raft_consensus.cc:3037] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:58:59.759351 18777 raft_consensus.cc:2454] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c395973ec204eea94b7fd8c4bc89c31 in term 1.
I20260925 14:58:59.759511 18643 raft_consensus.cc:2454] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c395973ec204eea94b7fd8c4bc89c31 in term 1.
I20260925 14:58:59.759521 18844 leader_election.cc:302] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7c395973ec204eea94b7fd8c4bc89c31, e4997f0ce4764d0b91e9546c18ced9ba; no voters:
I20260925 14:58:59.759635 18982 raft_consensus.cc:2781] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:58:59.759778 18982 raft_consensus.cc:686] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 1 LEADER]: Becoming Leader. State: Replica: 7c395973ec204eea94b7fd8c4bc89c31, State: Running, Role: LEADER
I20260925 14:58:59.759865 18982 consensus_queue.cc:237] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } }
I20260925 14:58:59.760672 18497 catalog_manager.cc:5495] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c395973ec204eea94b7fd8c4bc89c31 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "7c395973ec204eea94b7fd8c4bc89c31" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c395973ec204eea94b7fd8c4bc89c31" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42989 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 } health_report { overall_health: UNKNOWN } } }
I20260925 14:58:59.782454 18971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.780192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51020 (local address 127.8.13.195:42989)
0925 14:58:59.780248 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 14:58:59.780250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:58:59.780258 (+ 8us) server_negotiation.cc:407] Connection header received
0925 14:58:59.780318 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:58:59.780319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:58:59.780336 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:58:59.780388 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:58:59.780507 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.780984 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.781776 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:58:59.781966 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:58:59.782023 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:58:59.782160 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:58:59.782261 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:58:59.782318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 14:58:59.782336 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 14:58:59.782353 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 14:58:59.784293 18643 raft_consensus.cc:1261] T 545a4fd4c03241af9e2a13f4034b324a P 8097fb4d9fcf4ec9b83b19f2cecfbb53 [term 1 FOLLOWER]: Refusing update from remote peer 7c395973ec204eea94b7fd8c4bc89c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:59.784293 18777 raft_consensus.cc:1261] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 1 FOLLOWER]: Refusing update from remote peer 7c395973ec204eea94b7fd8c4bc89c31: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:58:59.784653 18982 consensus_queue.cc:1035] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4997f0ce4764d0b91e9546c18ced9ba" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:59.784818 18984 consensus_queue.cc:1035] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8097fb4d9fcf4ec9b83b19f2cecfbb53" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:58:59.787645 18988 mvcc.cc:205] Tried to move back new op lower bound from 7333266799753773056 to 7333266799657119744. Current Snapshot: MvccSnapshot[applied={T|T < 7333266799753773056}]
I20260925 14:58:59.787734 18989 mvcc.cc:205] Tried to move back new op lower bound from 7333266799753773056 to 7333266799657119744. Current Snapshot: MvccSnapshot[applied={T|T < 7333266799753773056}]
I20260925 14:58:59.789671 18498 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:59.789775 18498 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54640:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:58:59.790484 18498 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09e4a39b6a034e18976df18fae296661: Sending DeleteTablet for 3 replicas of tablet 545a4fd4c03241af9e2a13f4034b324a
I20260925 14:58:59.790784 18891 tablet_service.cc:1514] Processing DeleteTablet for tablet 545a4fd4c03241af9e2a13f4034b324a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:51012
I20260925 14:58:59.790825 18623 tablet_service.cc:1514] Processing DeleteTablet for tablet 545a4fd4c03241af9e2a13f4034b324a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:41580
I20260925 14:58:59.791026 18757 tablet_service.cc:1514] Processing DeleteTablet for tablet 545a4fd4c03241af9e2a13f4034b324a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:58:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46174
I20260925 14:58:59.791229 18999 tablet_replica.cc:330] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: stopping tablet replica
I20260925 14:58:59.791321 18999 raft_consensus.cc:2229] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:58:59.791320 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18560
I20260925 14:58:59.792135 18999 raft_consensus.cc:2258] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.792588 18999 ts_tablet_manager.cc:1905] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:59.792868 18997 tablet_replica.cc:330] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: stopping tablet replica
I20260925 14:58:59.792953 18997 raft_consensus.cc:2229] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:58:59.793033 18997 raft_consensus.cc:2258] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:58:59.793424 18997 ts_tablet_manager.cc:1905] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:58:59.794226 18999 ts_tablet_manager.cc:1918] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:59.794299 18999 log.cc:1196] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/545a4fd4c03241af9e2a13f4034b324a
I20260925 14:58:59.794549 18999 ts_tablet_manager.cc:1939] T 545a4fd4c03241af9e2a13f4034b324a P e4997f0ce4764d0b91e9546c18ced9ba: Deleting consensus metadata
I20260925 14:58:59.794875 18997 ts_tablet_manager.cc:1918] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:58:59.794937 18997 log.cc:1196] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/545a4fd4c03241af9e2a13f4034b324a
I20260925 14:58:59.795032 18482 catalog_manager.cc:4841] TS e4997f0ce4764d0b91e9546c18ced9ba (127.8.13.194:43921): tablet 545a4fd4c03241af9e2a13f4034b324a (table test-table [id=01c8357b1b614498b674b467e42eff34]) successfully deleted
I20260925 14:58:59.795113 18997 ts_tablet_manager.cc:1939] T 545a4fd4c03241af9e2a13f4034b324a P 7c395973ec204eea94b7fd8c4bc89c31: Deleting consensus metadata
I20260925 14:58:59.795617 18484 catalog_manager.cc:4841] TS 7c395973ec204eea94b7fd8c4bc89c31 (127.8.13.195:42989): tablet 545a4fd4c03241af9e2a13f4034b324a (table test-table [id=01c8357b1b614498b674b467e42eff34]) successfully deleted
W20260925 14:58:59.798298 18483 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:43613 (error 108)
I20260925 14:58:59.798556 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18694
I20260925 14:58:59.799154 18968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:58:59.798658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43613 (local address 127.0.0.1:41594)
0925 14:58:59.799055 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 14:58:59.799119 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371}
W20260925 14:58:59.800155 18483 catalog_manager.cc:4584] TS 8097fb4d9fcf4ec9b83b19f2cecfbb53 (127.8.13.193:43613): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 545a4fd4c03241af9e2a13f4034b324a: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43613: connect: Connection refused (error 111)
I20260925 14:58:59.803966 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18828
I20260925 14:58:59.808085 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 18466
2026-09-25T14:58:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1942 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[19003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[19003](info): set up 2 sockets
Sep 25 14:58:59 dist-test-slave-15hz krb5kdc[19003](info): commencing operation
krb5kdc: starting...
W20260925 14:59:00.756121 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:59:00 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangeradmin_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangeradmin_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangerlookup_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangerlookup_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
I20260925 14:59:00.918404 8247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxyjGoh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/postgres -l logfile start
2026-09-25 14:59:03.311 UTC [19033] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:59:03.312 UTC [19033] LOG: listening on IPv4 address "127.8.13.212", port 42829
2026-09-25 14:59:03.315 UTC [19033] LOG: listening on Unix socket "/tmp/.s.PGSQL.42829"
2026-09-25 14:59:03.336 UTC [19036] LOG: database system was shut down at 2026-09-25 14:59:01 UTC
2026-09-25 14:59:03.339 UTC [19033] LOG: database system is ready to accept connections
I20260925 14:59:04.232074 8247 mini_postgres.cc:96] Postgres bound to 42829
2026-09-25 14:59:04.237 UTC [19044] FATAL: database "slave" does not exist
127.8.13.212:42829 - accepting connections
I20260925 14:59:04.238163 8247 mini_ranger.cc:161] Starting Ranger...
I20260925 14:59:04.251713 8247 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:59:04.674598 8247 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:59:04.674665 8247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin
2026-09-25 14:59:04,945 [I] DB FLAVOR :POSTGRES
2026-09-25 14:59:04,945 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:59:04,945 [I] Checking connection..
2026-09-25 14:59:04,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:59:05,203 [I] Checking connection passed.
2026-09-25 14:59:05,204 [I] --------- Verifying version history table ---------
2026-09-25 14:59:05,204 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:05,453 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:05,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:05,682 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:05,682 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:59:05,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:59:05.915 UTC [19108] WARNING: there is no transaction in progress
2026-09-25 14:59:05,921 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:59:05,921 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:06,166 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:59:06,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:59:06,166 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:59:06,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:59:06,653 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-15hz.c.gcp-upstream.internal','N') ;"
2026-09-25 14:59:06,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:07,104 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:59:07,104 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:07,361 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:59:07,361 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:07,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:07,834 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:59:07,835 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:59:08.766 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.782 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.798 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.814 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.830 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.846 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.899 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.905 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.919 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.929 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.939 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.951 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.961 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.967 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.973 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:08.978 UTC [19216] WARNING: there is no transaction in progress
2026-09-25 14:59:09,565 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:59:09,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:59:09,814 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:59:09,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:10,074 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:59:10,074 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:10,317 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:59:10,317 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:10,557 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:10,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-15hz.c.gcp-upstream.internal';"
2026-09-25 14:59:11,045 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:59:11,045 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:59:11,283 [I] DB_PATCHES have already been applied
I20260925 14:59:11.288934 8247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxyjGoh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:59:11.289004 8247 mini_ranger.cc:193] Using host: 127.8.13.212
I20260925 14:59:11.291923 8247 mini_ranger.cc:227] Ranger admin URL: http://127.8.13.212:35313
Sep 25, 2026 2:59:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:59:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:59:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:59:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:59:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:59:12.287048 8247 mini_ranger.cc:160] Time spent starting Ranger: real 8.049s user 0.000s sys 0.008s
Sep 25, 2026 2:59:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 2:59:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 25, 2026 2:59:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 2:59:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 2:59:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 14:59:33.801097 8247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangerkms_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/rangerkms_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.13.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 14:59:33 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348373, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 14:59:33.883912 8247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 14:59:33.892297 8247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 14:59:34.650069 8247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 14:59:34.650156 8247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms
I20260925 14:59:34.650173 8247 mini_ranger_kms.cc:227] Using postgres at 127.8.13.212:42829
2026-09-25 14:59:34,805 [I] DB FLAVOR :POSTGRES
2026-09-25 14:59:34,805 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:59:34,806 [I] Checking connection
2026-09-25 14:59:34,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 14:59:35,044 [I] connection success
2026-09-25 14:59:35,044 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 14:59:35,044 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 14:59:35,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 14:59:35,297 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 14:59:35,297 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:59:35,297 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 14:59:35,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:42829/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 14:59:35,566 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 14:59:36.018435 8247 mini_ranger_kms.cc:310] Created kms service
I20260925 14:59:36.187850 8247 mini_ranger_kms.cc:326] Created kudu user
I20260925 14:59:36.229192 8247 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 14:59:36 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348373, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:36.645550 8247 mini_ranger_kms.cc:384] Added ranger policy
I20260925 14:59:36.645673 8247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 14:59:36.645696 8247 mini_ranger_kms.cc:240] Using host: 127.8.13.212
I20260925 14:59:36.648913 8247 mini_ranger_kms.cc:276] Ranger KMS PID: 19396
I20260925 14:59:36.649036 8247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.13.212:55419
Sep 25, 2026 2:59:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:37 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 2:59:37 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 2:59:38 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 2:59:38 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 14:59:38.679235 8247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.795s user 0.012s sys 0.126s
I20260925 14:59:38.679337 8247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 14:59:38.679410 8247 mini_ranger_kms.cc:401] 127.8.13.212:55419/kms/v1/keys
Sep 25 14:59:38 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348378, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 14:59:39 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348378, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
Sep 25 14:59:39 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348378, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
Sep 25 14:59:41 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 14:59:41.910914 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46369
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46369
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:55419/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:41.994540 19447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:41.994767 19447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:41.994794 19447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:41.996068 19447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 14:59:41.996114 19447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:41.996126 19447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:41.996137 19447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 14:59:41.996147 19447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 14:59:41.997706 19447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.13.212:55419/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46369
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:59:41.997936 19447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:41.998160 19447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:42.003011 19452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.003049 19453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.003031 19455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:42.003234 19447 server_base.cc:1034] running on GCE node
I20260925 14:59:42.003593 19447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:42.003908 19447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:42.005079 19447 hybrid_clock.cc:648] HybridClock initialized: now 1790348382005054 us; error 42 us; skew 500 ppm
Sep 25 14:59:42 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:42.010934 19447 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 14:59:42.011459 19447 webserver.cc:460] Webserver started at http://127.8.13.254:46383/ using document root <none> and password file <none>
I20260925 14:59:42.011716 19447 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:42.011761 19447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:42.011894 19447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:59:42 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:42.058017 19447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-15hz"
server_key: "ae502b7b8a6ca7a2249082575756b6dd"
server_key_iv: "96e469b073118173f7eab4444e964c38"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:42.058384 19447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-15hz"
server_key: "ae502b7b8a6ca7a2249082575756b6dd"
server_key_iv: "96e469b073118173f7eab4444e964c38"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:42.059835 19447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 14:59:42.072337 19462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:42.072646 19447 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:42.072769 19447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826"
format_stamp: "Formatted at 2026-09-25 14:59:42 on dist-test-slave-15hz"
server_key: "ae502b7b8a6ca7a2249082575756b6dd"
server_key_iv: "96e469b073118173f7eab4444e964c38"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:42.072896 19447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:42.087016 19447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:42.089056 19447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:42.089181 19447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:42.094069 19447 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:46369
I20260925 14:59:42.094117 19514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:46369 every 8 connection(s)
I20260925 14:59:42.094458 19447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 14:59:42.095266 19515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:42.096781 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 19447
I20260925 14:59:42.096880 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 14:59:42.097050 8247 external_mini_cluster.cc:1428] Setting key 847a0151a0468d880ebaa87d7d7c9cf7
I20260925 14:59:42.098642 19515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: Bootstrap starting.
I20260925 14:59:42.099445 19515 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:42.099843 19515 log.cc:824] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:42.100577 19515 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: No bootstrap required, opened a new log
I20260925 14:59:42.103116 19515 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } }
I20260925 14:59:42.103281 19515 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:42.103313 19515 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc0a74b8acc4eddb3d3dcde97bc4826, State: Initialized, Role: FOLLOWER
I20260925 14:59:42.103590 19515 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } }
I20260925 14:59:42.103763 19515 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 14:59:42.103819 19515 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 14:59:42.103864 19515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:42.104532 19515 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } }
I20260925 14:59:42.104667 19515 leader_election.cc:302] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4dc0a74b8acc4eddb3d3dcde97bc4826; no voters:
I20260925 14:59:42.104854 19515 leader_election.cc:288] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 14:59:42.104911 19520 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:42.105139 19520 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [term 1 LEADER]: Becoming Leader. State: Replica: 4dc0a74b8acc4eddb3d3dcde97bc4826, State: Running, Role: LEADER
I20260925 14:59:42.105242 19515 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 14:59:42.105275 19520 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } }
I20260925 14:59:42.105737 19520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } } }
I20260925 14:59:42.105875 19520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [sys.catalog]: This master's current role is: LEADER
Sep 25 14:59:42 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348381, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:59:42.106045 19520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4dc0a74b8acc4eddb3d3dcde97bc4826. Latest consensus state: current_term: 1 leader_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4dc0a74b8acc4eddb3d3dcde97bc4826" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46369 } } }
I20260925 14:59:42.106172 19520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826 [sys.catalog]: This master's current role is: LEADER
I20260925 14:59:42.106314 19528 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 14:59:42.108574 19528 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 14:59:42.108843 19515 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 14:59:42.113220 19518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:42.097562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32960 (local address 127.8.13.254:46369)
0925 14:59:42.098836 (+ 1274us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:42.098846 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:42.098874 (+ 28us) server_negotiation.cc:407] Connection header received
0925 14:59:42.099296 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:42.099303 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:42.099354 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:42.099487 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:42.100265 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.101756 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.102837 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:42.103521 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:42.103673 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:42.106584 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:42.106589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:42.106589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:42.106601 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:42.108468 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.109195 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.109198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.109199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.109296 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:42.109661 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:42.109662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:42.109663 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:42.111470 (+ 1807us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:42.111935 (+ 465us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:42.112076 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 14:59:42.112810 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1228,"thread_start_us":64,"threads_started":1}
I20260925 14:59:42.117698 19528 catalog_manager.cc:1347] Generated new cluster ID: 1d0b594316174a6da4c0407bf189474f
I20260925 14:59:42.117762 19528 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 14:59:42.133785 19528 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 14:59:42.134639 19528 catalog_manager.cc:1504] Loading token signing keys...
I20260925 14:59:42.137838 19528 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: Generated new TSK 0
I20260925 14:59:42.138212 19528 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 14:59:42.870744 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:55419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:42.979390 19560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:42.979713 19560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:42.979750 19560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:42.981408 19560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:42.981469 19560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:42.981514 19560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 14:59:42.983153 19560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.13.212:55419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:59:42.983376 19560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:42.983631 19560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 14:59:42.990226 19560 server_base.cc:1034] running on GCE node
W20260925 14:59:42.990226 19568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.990396 19565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:42.990487 19566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:42.990729 19560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:42.990988 19560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:42.992166 19560 hybrid_clock.cc:648] HybridClock initialized: now 1790348382992143 us; error 32 us; skew 500 ppm
Sep 25 14:59:42 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:42.995481 19560 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 14:59:42.996189 19560 webserver.cc:460] Webserver started at http://127.8.13.193:33733/ using document root <none> and password file <none>
I20260925 14:59:42.996450 19560 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:42.996510 19560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:42.996683 19560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:59:42 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:43.015003 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "af54c3a58b7e489091b8a538893bb351"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "ea4efc0bb4f5cff26c4a7cecf55c99a0"
server_key_iv: "65f817538beef1f5c332c40c7c0d3bc6"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.015457 19560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "af54c3a58b7e489091b8a538893bb351"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "ea4efc0bb4f5cff26c4a7cecf55c99a0"
server_key_iv: "65f817538beef1f5c332c40c7c0d3bc6"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.017323 19560 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 14:59:43.037533 19577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.037811 19560 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.001s sys 0.000s
I20260925 14:59:43.037962 19560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "af54c3a58b7e489091b8a538893bb351"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "ea4efc0bb4f5cff26c4a7cecf55c99a0"
server_key_iv: "65f817538beef1f5c332c40c7c0d3bc6"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.038138 19560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:43.063488 19560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:43.065939 19560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:43.066093 19560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:43.066352 19560 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:43.066717 19560 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:43.066759 19560 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.066788 19560 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:43.066846 19560 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.073689 19560 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:41191
I20260925 14:59:43.074188 19560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 14:59:43.077467 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 19560
I20260925 14:59:43.077598 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 14:59:43.077804 8247 external_mini_cluster.cc:1428] Setting key c064d6219edfe5d8466056c6df76b38a
I20260925 14:59:43.078681 19690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:41191 every 8 connection(s)
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348382, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:59:43.112861 19693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.076991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46369 (local address 127.8.13.193:43615)
0925 14:59:43.090167 (+ 13176us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.090250 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.090962 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.097271 (+ 6309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.097277 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.097541 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:43.097850 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.097856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.098875 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.098876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.100154 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.100158 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.100713 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.100715 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.100874 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.101491 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:43.101501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:43.108864 (+ 7363us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:43.111052 (+ 2188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.111066 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.111069 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.111414 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.111689 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.111690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.111690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.111837 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.112123 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:43.112131 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:43.112331 (+ 200us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.112428 (+ 97us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.112609 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13104,"thread_start_us":77,"threads_started":1}
I20260925 14:59:43.113690 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.077309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43615 (local address 127.8.13.254:46369)
0925 14:59:43.096987 (+ 19678us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.096989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.097002 (+ 13us) server_negotiation.cc:407] Connection header received
0925 14:59:43.097030 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.097031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.097044 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.097143 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.097981 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.098750 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.100260 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.100559 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.102279 (+ 1720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.109066 (+ 6787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.109069 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.109069 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.109079 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.110936 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.111505 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.111506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.111507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.111613 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.111904 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.111904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.111905 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.112046 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.113451 (+ 1405us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.113496 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.113592 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19643,"thread_start_us":76,"threads_started":1}
I20260925 14:59:43.114274 19691 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46369
I20260925 14:59:43.114372 19691 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:43.114562 19691 heartbeater.cc:511] Master 127.8.13.254:46369 requested a full tablet report, sending...
I20260925 14:59:43.115072 19479 ts_manager.cc:194] Registered new tserver with Master: af54c3a58b7e489091b8a538893bb351 (127.8.13.193:41191)
I20260925 14:59:43.115955 19479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:43615
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 14:59:43.131852 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:55419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260925 14:59:43.275413 19699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:43.275864 19699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:43.275943 19699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:43.277892 19699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:43.278151 19699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:43.278252 19699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 14:59:43.280340 19699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.13.212:55419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:59:43.280625 19699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:43.280864 19699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:43.285813 19707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:43.285884 19704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:43.285817 19705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:43.286708 19699 server_base.cc:1034] running on GCE node
I20260925 14:59:43.286901 19699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:43.287120 19699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:43.288254 19699 hybrid_clock.cc:648] HybridClock initialized: now 1790348383288239 us; error 32 us; skew 500 ppm
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:43.290832 19699 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 14:59:43.291261 19699 webserver.cc:460] Webserver started at http://127.8.13.194:40691/ using document root <none> and password file <none>
I20260925 14:59:43.291486 19699 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:43.291558 19699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:43.291689 19699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:43.304140 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "2d454813c9644bd9918b21741a61a2ee"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "4ccfbb81c983e5aaa1fac477984d8138"
server_key_iv: "a6b478a56eb09f622d2a2432b867a4fd"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.304543 19699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "2d454813c9644bd9918b21741a61a2ee"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "4ccfbb81c983e5aaa1fac477984d8138"
server_key_iv: "a6b478a56eb09f622d2a2432b867a4fd"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.306282 19699 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 14:59:43.321708 19714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.321916 19699 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 14:59:43.322016 19699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "2d454813c9644bd9918b21741a61a2ee"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "4ccfbb81c983e5aaa1fac477984d8138"
server_key_iv: "a6b478a56eb09f622d2a2432b867a4fd"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.322175 19699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:43.333184 19699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:43.335216 19699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:43.335433 19699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:43.335796 19699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:43.336236 19699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:43.336326 19699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.336400 19699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:43.336454 19699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.343205 19699 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:32815
I20260925 14:59:43.344709 19699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 14:59:43.345860 19828 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:32815 every 8 connection(s)
I20260925 14:59:43.350373 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 19699
I20260925 14:59:43.350473 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 14:59:43.350669 8247 external_mini_cluster.cc:1428] Setting key 66e591abe3a9cf808bd0ee5db267ab12
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:59:43.359956 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.347004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56747 (local address 127.8.13.254:46369)
0925 14:59:43.347070 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.347072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.347703 (+ 631us) server_negotiation.cc:407] Connection header received
0925 14:59:43.348401 (+ 698us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.348402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.348419 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.348450 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.349286 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.350137 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.353980 (+ 3843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.354310 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.354351 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.356949 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.356951 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.356951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.356960 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.358509 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.358990 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.358992 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.358992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.359053 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.359280 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.359281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.359282 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.359394 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.359447 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.359768 (+ 321us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.359861 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 14:59:43.360172 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.347141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46369 (local address 127.8.13.194:56747)
0925 14:59:43.347548 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.347604 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.348276 (+ 672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.348639 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.348644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.348887 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:43.349174 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.349182 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.350980 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.350981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.353880 (+ 2899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.353883 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.354443 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.354443 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.354594 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.355154 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:43.355164 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:43.356837 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:43.358635 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.358643 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.358646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.358896 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.359109 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.359110 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.359110 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.359213 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.359447 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:43.359453 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:43.359636 (+ 183us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.359724 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.359873 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":90,"threads_started":1}
I20260925 14:59:43.360674 19829 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46369
I20260925 14:59:43.360776 19829 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:43.360982 19829 heartbeater.cc:511] Master 127.8.13.254:46369 requested a full tablet report, sending...
I20260925 14:59:43.361387 19479 ts_manager.cc:194] Registered new tserver with Master: 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815)
I20260925 14:59:43.361848 19479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:56747
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 14:59:43.385988 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:55419/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:45623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 14:59:43.509203 19834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 14:59:43.509542 19834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 14:59:43.509626 19834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 14:59:43.511376 19834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 14:59:43.511693 19834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 14:59:43.511795 19834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 14:59:43.513979 19834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:45623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.13.212:55419/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46369
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 14:59:43.514451 19834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 14:59:43.514768 19834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 14:59:43.519946 19842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:43.520357 19840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 14:59:43.520603 19839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 14:59:43.524729 19834 server_base.cc:1034] running on GCE node
I20260925 14:59:43.525077 19834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 14:59:43.525463 19834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 14:59:43.526611 19834 hybrid_clock.cc:648] HybridClock initialized: now 1790348383526598 us; error 26 us; skew 500 ppm
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 14:59:43.529556 19834 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 14:59:43.530002 19834 webserver.cc:460] Webserver started at http://127.8.13.195:33461/ using document root <none> and password file <none>
I20260925 14:59:43.530231 19834 fs_manager.cc:359] Metadata directory not provided
I20260925 14:59:43.530282 19834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 14:59:43.530416 19834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:43.551223 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "6bd092d3787145baac3602f18c4e811d"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "f8423694ed97da6a5d78c900a70361d2"
server_key_iv: "b7b75719194bb53b582f9bc144e4f731"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.551726 19834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "6bd092d3787145baac3602f18c4e811d"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "f8423694ed97da6a5d78c900a70361d2"
server_key_iv: "b7b75719194bb53b582f9bc144e4f731"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.553603 19834 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260925 14:59:43.569273 19849 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.569525 19834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 14:59:43.569631 19834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "6bd092d3787145baac3602f18c4e811d"
format_stamp: "Formatted at 2026-09-25 14:59:43 on dist-test-slave-15hz"
server_key: "f8423694ed97da6a5d78c900a70361d2"
server_key_iv: "b7b75719194bb53b582f9bc144e4f731"
server_key_version: "kuduclusterkey@0"
I20260925 14:59:43.569743 19834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 14:59:43.592285 19834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 14:59:43.598418 19834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 14:59:43.599009 19834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 14:59:43.599318 19834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 14:59:43.599725 19834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 14:59:43.599807 19834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.599871 19834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 14:59:43.599987 19834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 14:59:43.610723 19834 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:39257
I20260925 14:59:43.610805 19962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:39257 every 8 connection(s)
I20260925 14:59:43.611232 19834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 14:59:43.613086 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 19834
I20260925 14:59:43.613186 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 14:59:43.613382 8247 external_mini_cluster.cc:1428] Setting key d2681cbec7bdf0407752e32a8d294bf8
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 14:59:43.624769 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.612068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:49027 (local address 127.8.13.254:46369)
0925 14:59:43.612111 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.612113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.614478 (+ 2365us) server_negotiation.cc:407] Connection header received
0925 14:59:43.615177 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.615179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.615191 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.615218 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.616105 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.616907 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.618409 (+ 1502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.618723 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.618767 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.621410 (+ 2643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.621412 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.621412 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.621419 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.623061 (+ 1642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.623599 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.623601 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.623601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.623663 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.623978 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.623979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.623979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.624115 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.624183 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.624551 (+ 368us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.624662 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 14:59:43.625309 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.612050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46369 (local address 127.8.13.195:49027)
0925 14:59:43.614347 (+ 2297us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.614399 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.615056 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.615406 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.615411 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.615703 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0925 14:59:43.615993 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.616000 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.617025 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.617026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.618289 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.618292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.618857 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.618858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.619016 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.619685 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 14:59:43.619696 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 14:59:43.621304 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 14:59:43.623152 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.623159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.623161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.623474 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.623732 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 14:59:43.623733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 14:59:43.623734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 14:59:43.623888 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 14:59:43.624183 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 14:59:43.624188 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 14:59:43.624361 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.624884 (+ 523us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.625068 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"thread_start_us":82,"threads_started":1}
I20260925 14:59:43.626179 19963 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46369
I20260925 14:59:43.626287 19963 heartbeater.cc:464] Registering TS with master...
I20260925 14:59:43.626492 19963 heartbeater.cc:511] Master 127.8.13.254:46369 requested a full tablet report, sending...
I20260925 14:59:43.626881 19479 ts_manager.cc:194] Registered new tserver with Master: 6bd092d3787145baac3602f18c4e811d (127.8.13.195:39257)
I20260925 14:59:43.627403 19479 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:49027
I20260925 14:59:43.636143 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 14:59:43.646325 19694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.637878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32966 (local address 127.8.13.254:46369)
0925 14:59:43.637936 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.637939 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.639497 (+ 1558us) server_negotiation.cc:407] Connection header received
0925 14:59:43.639640 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.639641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.639654 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.639688 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 14:59:43.640567 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.641257 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.642066 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.642272 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.642335 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.643718 (+ 1383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 14:59:43.643720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 14:59:43.643720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 14:59:43.643727 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 14:59:43.645211 (+ 1484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.645551 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.645553 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.645553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.645611 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 14:59:43.645788 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 14:59:43.645789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 14:59:43.645790 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 14:59:43.645911 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 14:59:43.646003 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.646145 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.646223 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 14:59:43.648483 19479 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32966:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 14:59:43.650187 19479 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 14:59:43.658545 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.652642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39257 (local address 127.0.0.1:38228)
0925 14:59:43.653768 (+ 1126us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.653778 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.653819 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.654150 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.654151 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.654154 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.654204 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.654206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.655780 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.655781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.656848 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.656851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.658208 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.658210 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.658308 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.658326 (+ 18us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.658395 (+ 69us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.658451 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1109,"mutex_wait_us":4,"thread_start_us":88,"threads_started":1}
I20260925 14:59:43.659384 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.653024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38228 (local address 127.8.13.195:39257)
0925 14:59:43.653936 (+ 912us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.653941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.653950 (+ 9us) server_negotiation.cc:407] Connection header received
0925 14:59:43.653985 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.653989 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.654003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.654077 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.654281 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.655684 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.656981 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.658107 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.659134 (+ 1027us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.659221 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.659249 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.659291 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":106,"threads_started":1}
I20260925 14:59:43.660270 19890 tablet_service.cc:1467] Processing CreateTablet for tablet cb068a0c1b334aebb45ec0d50143c072 (DEFAULT_TABLE table=test-table [id=a9e0e5f1c3d841dd8954586a6f90b9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:43.660692 19890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb068a0c1b334aebb45ec0d50143c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:43.661921 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.652641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41191 (local address 127.0.0.1:41080)
0925 14:59:43.653267 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.653502 (+ 235us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.653640 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.653932 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.653935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.653938 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.654010 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.654012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.655342 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.655343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.657502 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.657505 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.661157 (+ 3652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.661159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.661221 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.661226 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.661280 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.661820 (+ 540us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":93,"threads_started":1}
I20260925 14:59:43.662257 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.653005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.8.13.193:41191)
0925 14:59:43.653117 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.653122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.653613 (+ 491us) server_negotiation.cc:407] Connection header received
0925 14:59:43.653740 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.653742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.653761 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.653837 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.654097 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.655242 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.658044 (+ 2802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.660979 (+ 2935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.661058 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.661119 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.662122 (+ 1003us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.662177 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":58,"threads_started":1}
I20260925 14:59:43.664278 19625 tablet_service.cc:1467] Processing CreateTablet for tablet cb068a0c1b334aebb45ec0d50143c072 (DEFAULT_TABLE table=test-table [id=a9e0e5f1c3d841dd8954586a6f90b9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:43.664630 19625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb068a0c1b334aebb45ec0d50143c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:43.666973 19981 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Bootstrap starting.
I20260925 14:59:43.669847 19981 tablet_bootstrap.cc:654] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:43.670223 19981 log.cc:824] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:43.670831 19982 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Bootstrap starting.
I20260925 14:59:43.671028 19981 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: No bootstrap required, opened a new log
I20260925 14:59:43.671089 19981 ts_tablet_manager.cc:1397] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260925 14:59:43.672070 19982 tablet_bootstrap.cc:654] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:43.672362 19982 log.cc:824] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:43.675464 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.652875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:32815 (local address 127.0.0.1:60312)
0925 14:59:43.659000 (+ 6125us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.659011 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.659050 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.669040 (+ 9990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.669051 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.669053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.669107 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.669109 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.672605 (+ 3496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.672606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.673766 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.673769 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.675216 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.675217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.675290 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.675296 (+ 6us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.675324 (+ 28us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.675380 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6113,"thread_start_us":64,"threads_started":1}
I20260925 14:59:43.675923 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.653224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60312 (local address 127.8.13.194:32815)
0925 14:59:43.668844 (+ 15620us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.668849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.668861 (+ 12us) server_negotiation.cc:407] Connection header received
0925 14:59:43.668911 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.668913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.668931 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.669184 (+ 253us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.669263 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.670304 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.673914 (+ 3610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.675131 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.675655 (+ 524us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.675757 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.675784 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.675826 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15572,"thread_start_us":82,"threads_started":1}
I20260925 14:59:43.676630 19744 tablet_service.cc:1467] Processing CreateTablet for tablet cb068a0c1b334aebb45ec0d50143c072 (DEFAULT_TABLE table=test-table [id=a9e0e5f1c3d841dd8954586a6f90b9a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 14:59:43.677017 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cb068a0c1b334aebb45ec0d50143c072. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 14:59:43.678683 19981 raft_consensus.cc:348] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.679003 19981 raft_consensus.cc:374] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:43.679093 19981 raft_consensus.cc:729] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af54c3a58b7e489091b8a538893bb351, State: Initialized, Role: FOLLOWER
I20260925 14:59:43.679286 19981 consensus_queue.cc:260] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.679698 19981 ts_tablet_manager.cc:1428] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20260925 14:59:43.680127 19691 heartbeater.cc:503] Master 127.8.13.254:46369 was elected leader, sending a full tablet report...
I20260925 14:59:43.681427 19986 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Bootstrap starting.
I20260925 14:59:43.682425 19986 tablet_bootstrap.cc:654] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Neither blocks nor log segments found. Creating new log.
I20260925 14:59:43.682688 19986 log.cc:824] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Log is configured to *not* fsync() on all Append() calls
I20260925 14:59:43.683512 19982 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: No bootstrap required, opened a new log
I20260925 14:59:43.683674 19982 ts_tablet_manager.cc:1397] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260925 14:59:43.685797 19982 raft_consensus.cc:348] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.686043 19982 raft_consensus.cc:374] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:43.686094 19982 raft_consensus.cc:729] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bd092d3787145baac3602f18c4e811d, State: Initialized, Role: FOLLOWER
I20260925 14:59:43.686215 19982 consensus_queue.cc:260] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.686482 19982 ts_tablet_manager.cc:1428] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260925 14:59:43.687042 19963 heartbeater.cc:503] Master 127.8.13.254:46369 was elected leader, sending a full tablet report...
I20260925 14:59:43.687939 19986 tablet_bootstrap.cc:492] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: No bootstrap required, opened a new log
I20260925 14:59:43.688002 19986 ts_tablet_manager.cc:1397] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260925 14:59:43.689740 19986 raft_consensus.cc:348] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.689867 19986 raft_consensus.cc:374] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 14:59:43.689908 19986 raft_consensus.cc:729] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d454813c9644bd9918b21741a61a2ee, State: Initialized, Role: FOLLOWER
I20260925 14:59:43.690028 19986 consensus_queue.cc:260] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.690290 19986 ts_tablet_manager.cc:1428] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 14:59:43.690414 19829 heartbeater.cc:503] Master 127.8.13.254:46369 was elected leader, sending a full tablet report...
Sep 25 14:59:43 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
W20260925 14:59:43.827447 19692 tablet.cc:2369] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:43.847800 19830 tablet.cc:2369] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260925 14:59:43.862200 19964 tablet.cc:2369] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 14:59:43.922906 19985 raft_consensus.cc:482] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:43.923065 19985 raft_consensus.cc:504] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.923431 19985 leader_election.cc:288] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6bd092d3787145baac3602f18c4e811d (127.8.13.195:39257), 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815)
I20260925 14:59:43.928683 19988 raft_consensus.cc:482] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 14:59:43.928804 19988 raft_consensus.cc:504] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.929165 19988 leader_election.cc:288] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af54c3a58b7e489091b8a538893bb351 (127.8.13.193:41191), 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815)
I20260925 14:59:43.931915 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.924389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:52339 (local address 127.8.13.195:39257)
0925 14:59:43.924490 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.924492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.924608 (+ 116us) server_negotiation.cc:407] Connection header received
0925 14:59:43.924741 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.924742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.924753 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.924781 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.924990 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.925780 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.931009 (+ 5229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.931558 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.931602 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.931611 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.931806 (+ 195us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.931843 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":80}
I20260925 14:59:43.932312 19991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.923736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:39257 (local address 127.8.13.193:52339)
0925 14:59:43.924092 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.924545 (+ 453us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.924667 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.924843 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.924844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.924845 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.924926 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.924928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.925924 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.925924 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.927172 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.927175 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.931662 (+ 4487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.931662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.931730 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.931734 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.932188 (+ 454us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.932230 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":92,"threads_started":1}
I20260925 14:59:43.932575 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.929402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41191 (local address 127.8.13.195:58811)
0925 14:59:43.929477 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.929488 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.929528 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.929702 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.929703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.929705 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.929756 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.929758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.930611 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.930612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.931722 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.931724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.932359 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.932360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.932436 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.932439 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.932478 (+ 39us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.932516 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260925 14:59:43.932864 19917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "af54c3a58b7e489091b8a538893bb351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd092d3787145baac3602f18c4e811d" is_pre_election: true
I20260925 14:59:43.932967 19917 raft_consensus.cc:2454] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af54c3a58b7e489091b8a538893bb351 in term 0.
I20260925 14:59:43.933230 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.929440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:58811 (local address 127.8.13.193:41191)
0925 14:59:43.929593 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.929595 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.929601 (+ 6us) server_negotiation.cc:407] Connection header received
0925 14:59:43.929620 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.929620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.929629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.929668 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.929817 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.930521 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.931799 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.932263 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.932322 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.932329 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.933135 (+ 806us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.933171 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260925 14:59:43.933490 19645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "6bd092d3787145baac3602f18c4e811d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af54c3a58b7e489091b8a538893bb351" is_pre_election: true
I20260925 14:59:43.933583 19580 leader_election.cc:302] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd092d3787145baac3602f18c4e811d, af54c3a58b7e489091b8a538893bb351; no voters:
I20260925 14:59:43.933622 19645 raft_consensus.cc:2454] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd092d3787145baac3602f18c4e811d in term 0.
I20260925 14:59:43.933884 19985 raft_consensus.cc:2781] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:43.933931 19985 raft_consensus.cc:482] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:43.933951 19985 raft_consensus.cc:3037] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.934836 19985 raft_consensus.cc:504] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.934957 19985 leader_election.cc:288] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [CANDIDATE]: Term 1 election: Requested vote from peers 6bd092d3787145baac3602f18c4e811d (127.8.13.195:39257), 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815)
I20260925 14:59:43.935212 19851 leader_election.cc:302] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6bd092d3787145baac3602f18c4e811d, af54c3a58b7e489091b8a538893bb351; no voters:
I20260925 14:59:43.935356 19988 raft_consensus.cc:2781] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 14:59:43.935429 19988 raft_consensus.cc:482] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 14:59:43.935470 19988 raft_consensus.cc:3037] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.935930 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.924446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:49541 (local address 127.8.13.194:32815)
0925 14:59:43.924524 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.924526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.927294 (+ 2768us) server_negotiation.cc:407] Connection header received
0925 14:59:43.927388 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.927389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.927401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.927429 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.927683 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.928382 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.932725 (+ 4343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.933237 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.933284 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.933293 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.935813 (+ 2520us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.935866 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 14:59:43.936316 19988 raft_consensus.cc:504] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.936441 19988 leader_election.cc:288] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [CANDIDATE]: Term 1 election: Requested vote from peers af54c3a58b7e489091b8a538893bb351 (127.8.13.193:41191), 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815)
I20260925 14:59:43.936515 19917 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "af54c3a58b7e489091b8a538893bb351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bd092d3787145baac3602f18c4e811d"
I20260925 14:59:43.936575 19917 raft_consensus.cc:2379] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate af54c3a58b7e489091b8a538893bb351 in current term 1: Already voted for candidate 6bd092d3787145baac3602f18c4e811d in this term.
I20260925 14:59:43.938305 19645 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "6bd092d3787145baac3602f18c4e811d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af54c3a58b7e489091b8a538893bb351"
I20260925 14:59:43.938416 19645 raft_consensus.cc:2379] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 6bd092d3787145baac3602f18c4e811d in current term 1: Already voted for candidate af54c3a58b7e489091b8a538893bb351 in this term.
I20260925 14:59:43.939756 19965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.929523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:32815 (local address 127.8.13.195:50041)
0925 14:59:43.932639 (+ 3116us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.932647 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.932679 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.936180 (+ 3501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.936181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.936183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.936218 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.936220 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.937052 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.937053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.938142 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.938145 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.939497 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.939498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.939596 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.939599 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.939644 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.939682 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3105,"thread_start_us":112,"threads_started":1}
I20260925 14:59:43.940188 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.929818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:50041 (local address 127.8.13.194:32815)
0925 14:59:43.936013 (+ 6195us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:43.936015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:43.936022 (+ 7us) server_negotiation.cc:407] Connection header received
0925 14:59:43.936042 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:43.936043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:43.936053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:43.936107 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 14:59:43.936290 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.936919 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.938829 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.939389 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.939456 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.939466 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:43.940077 (+ 611us) server_negotiation.cc:299] Negotiation successful
0925 14:59:43.940120 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6179,"thread_start_us":55,"threads_started":1}
I20260925 14:59:43.940438 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "6bd092d3787145baac3602f18c4e811d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d454813c9644bd9918b21741a61a2ee" is_pre_election: true
I20260925 14:59:43.940603 19783 raft_consensus.cc:2454] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6bd092d3787145baac3602f18c4e811d in term 0.
I20260925 14:59:43.940621 19990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:43.923889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:32815 (local address 127.8.13.193:49541)
0925 14:59:43.924121 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:43.927254 (+ 3133us) client_negotiation.cc:174] Beginning negotiation
0925 14:59:43.927321 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 14:59:43.927516 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 14:59:43.927516 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 14:59:43.927518 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 14:59:43.927613 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.927614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.928465 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.928466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.930914 (+ 2448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 14:59:43.930917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 14:59:43.935405 (+ 4488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 14:59:43.935407 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 14:59:43.935484 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:43.935488 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 14:59:43.940464 (+ 4976us) client_negotiation.cc:240] Negotiation successful
0925 14:59:43.940521 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":44,"threads_started":1}
I20260925 14:59:43.940701 19782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "6bd092d3787145baac3602f18c4e811d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d454813c9644bd9918b21741a61a2ee"
I20260925 14:59:43.940745 19782 raft_consensus.cc:3037] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 0 FOLLOWER]: Advancing to term 1
I20260925 14:59:43.940871 19781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "af54c3a58b7e489091b8a538893bb351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d454813c9644bd9918b21741a61a2ee"
I20260925 14:59:43.941157 19580 leader_election.cc:302] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af54c3a58b7e489091b8a538893bb351; no voters: 2d454813c9644bd9918b21741a61a2ee, 6bd092d3787145baac3602f18c4e811d
I20260925 14:59:43.941311 19783 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cb068a0c1b334aebb45ec0d50143c072" candidate_uuid: "af54c3a58b7e489091b8a538893bb351" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2d454813c9644bd9918b21741a61a2ee" is_pre_election: true
I20260925 14:59:43.941499 19985 raft_consensus.cc:2726] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260925 14:59:43.941571 19782 raft_consensus.cc:2454] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6bd092d3787145baac3602f18c4e811d in term 1.
I20260925 14:59:43.941794 19852 leader_election.cc:302] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2d454813c9644bd9918b21741a61a2ee, 6bd092d3787145baac3602f18c4e811d; no voters: af54c3a58b7e489091b8a538893bb351
I20260925 14:59:43.941890 19988 raft_consensus.cc:2781] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 14:59:43.942032 19988 raft_consensus.cc:686] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 LEADER]: Becoming Leader. State: Replica: 6bd092d3787145baac3602f18c4e811d, State: Running, Role: LEADER
I20260925 14:59:43.942108 19988 consensus_queue.cc:237] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } }
I20260925 14:59:43.942929 19479 catalog_manager.cc:5495] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6bd092d3787145baac3602f18c4e811d (127.8.13.195). New cstate: current_term: 1 leader_uuid: "6bd092d3787145baac3602f18c4e811d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6bd092d3787145baac3602f18c4e811d" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 39257 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 } health_report { overall_health: UNKNOWN } } }
I20260925 14:59:44.017168 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:44.013894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.8.13.195:39257)
0925 14:59:44.013934 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 14:59:44.013936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 14:59:44.013946 (+ 10us) server_negotiation.cc:407] Connection header received
0925 14:59:44.014021 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 14:59:44.014022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 14:59:44.014043 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 14:59:44.014112 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 14:59:44.014276 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:44.014989 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:44.016106 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 14:59:44.016379 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 14:59:44.016428 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 14:59:44.016609 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 14:59:44.016750 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 14:59:44.016817 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 14:59:44.017045 (+ 228us) server_negotiation.cc:299] Negotiation successful
0925 14:59:44.017068 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 14:59:44.020159 19782 raft_consensus.cc:1261] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 1 FOLLOWER]: Refusing update from remote peer 6bd092d3787145baac3602f18c4e811d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:44.020540 19994 consensus_queue.cc:1035] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2d454813c9644bd9918b21741a61a2ee" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 32815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:44.022344 19645 raft_consensus.cc:1261] T cb068a0c1b334aebb45ec0d50143c072 P af54c3a58b7e489091b8a538893bb351 [term 1 FOLLOWER]: Refusing update from remote peer 6bd092d3787145baac3602f18c4e811d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 14:59:44.022935 19994 consensus_queue.cc:1035] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [LEADER]: Connected to new peer: Peer: permanent_uuid: "af54c3a58b7e489091b8a538893bb351" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 14:59:44.027738 19997 mvcc.cc:205] Tried to move back new op lower bound from 7333266980943540224 to 7333266980627419136. Current Snapshot: MvccSnapshot[applied={T|T < 7333266980943540224}]
I20260925 14:59:44.028790 20001 mvcc.cc:205] Tried to move back new op lower bound from 7333266980943540224 to 7333266980627419136. Current Snapshot: MvccSnapshot[applied={T|T < 7333266980943540224}]
I20260925 14:59:44.031952 19479 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:44.032078 19479 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32966:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 14:59:44.032881 19479 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4dc0a74b8acc4eddb3d3dcde97bc4826: Sending DeleteTablet for 3 replicas of tablet cb068a0c1b334aebb45ec0d50143c072
I20260925 14:59:44.033268 19744 tablet_service.cc:1514] Processing DeleteTablet for tablet cb068a0c1b334aebb45ec0d50143c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:44 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:60312
I20260925 14:59:44.033401 19625 tablet_service.cc:1514] Processing DeleteTablet for tablet cb068a0c1b334aebb45ec0d50143c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:44 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:41080
I20260925 14:59:44.033622 19890 tablet_service.cc:1514] Processing DeleteTablet for tablet cb068a0c1b334aebb45ec0d50143c072 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 14:59:44 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:38228
I20260925 14:59:44.038264 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 19560
I20260925 14:59:44.042109 20008 tablet_replica.cc:330] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: stopping tablet replica
I20260925 14:59:44.042814 20008 raft_consensus.cc:2229] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 LEADER]: Raft consensus shutting down.
I20260925 14:59:44.043073 20008 raft_consensus.cc:2258] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:44.043601 20008 ts_tablet_manager.cc:1905] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 14:59:44.045257 20006 tablet_replica.cc:330] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: stopping tablet replica
I20260925 14:59:44.045434 20006 raft_consensus.cc:2229] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 14:59:44.045598 20006 raft_consensus.cc:2258] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 14:59:44.052744 20006 ts_tablet_manager.cc:1905] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 25 14:59:44 dist-test-slave-15hz krb5kdc[19003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348383, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 14:59:44.055544 20006 ts_tablet_manager.cc:1918] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:44.055517 20008 ts_tablet_manager.cc:1918] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 14:59:44.055644 20008 log.cc:1196] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/cb068a0c1b334aebb45ec0d50143c072
I20260925 14:59:44.055756 20006 log.cc:1196] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/cb068a0c1b334aebb45ec0d50143c072
I20260925 14:59:44.055985 20008 ts_tablet_manager.cc:1939] T cb068a0c1b334aebb45ec0d50143c072 P 6bd092d3787145baac3602f18c4e811d: Deleting consensus metadata
I20260925 14:59:44.056090 20006 ts_tablet_manager.cc:1939] T cb068a0c1b334aebb45ec0d50143c072 P 2d454813c9644bd9918b21741a61a2ee: Deleting consensus metadata
I20260925 14:59:44.056552 19465 catalog_manager.cc:4841] TS 2d454813c9644bd9918b21741a61a2ee (127.8.13.194:32815): tablet cb068a0c1b334aebb45ec0d50143c072 (table test-table [id=a9e0e5f1c3d841dd8954586a6f90b9a7]) successfully deleted
I20260925 14:59:44.056655 19465 catalog_manager.cc:4841] TS 6bd092d3787145baac3602f18c4e811d (127.8.13.195:39257): tablet cb068a0c1b334aebb45ec0d50143c072 (table test-table [id=a9e0e5f1c3d841dd8954586a6f90b9a7]) successfully deleted
W20260925 14:59:44.058573 19464 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:41191 (error 108)
I20260925 14:59:44.066740 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 19699
I20260925 14:59:44.073933 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 19834
I20260925 14:59:44.076491 19976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 14:59:44.076352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41191 (local address 127.0.0.1:41092)
0925 14:59:44.076405 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 14:59:44.076446 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 14:59:44.076731 19464 catalog_manager.cc:4584] TS af54c3a58b7e489091b8a538893bb351 (127.8.13.193:41191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cb068a0c1b334aebb45ec0d50143c072: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41191: connect: Connection refused (error 111)
I20260925 14:59:44.080905 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 19447
I20260925 14:59:44.086553 8247 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 14:59:44.430603 8247 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 14:59:44.430716 8247 mini_ranger.cc:66] Stopping Ranger...
I20260925 14:59:44.765038 8247 mini_ranger.cc:68] Stopped Ranger
2026-09-25 14:59:44.765 UTC [19033] LOG: received smart shutdown request
2026-09-25 14:59:44.768 UTC [19033] LOG: background worker "logical replication launcher" (PID 19042) exited with exit code 1
2026-09-25 14:59:44.768 UTC [19037] LOG: shutting down
2026-09-25 14:59:44.919 UTC [19033] LOG: database system is shut down
2026-09-25T14:59:44Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45198 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 14:59:45 dist-test-slave-15hz krb5kdc[20020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 14:59:45 dist-test-slave-15hz krb5kdc[20020](info): set up 2 sockets
Sep 25 14:59:45 dist-test-slave-15hz krb5kdc[20020](info): commencing operation
krb5kdc: starting...
W20260925 14:59:45.967248 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 14:59:46 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348386, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T14:59:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T14:59:46Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangeradmin_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangeradmin_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangerlookup_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangerlookup_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
I20260925 14:59:46.120421 8247 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskxyjGoh/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/postgres -l logfile start
2026-09-25 14:59:48.474 UTC [20050] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-25 14:59:48.474 UTC [20050] LOG: listening on IPv4 address "127.8.13.212", port 47585
2026-09-25 14:59:48.477 UTC [20050] LOG: listening on Unix socket "/tmp/.s.PGSQL.47585"
2026-09-25 14:59:48.504 UTC [20053] LOG: database system was shut down at 2026-09-25 14:59:47 UTC
2026-09-25 14:59:48.508 UTC [20050] LOG: database system is ready to accept connections
I20260925 14:59:49.375082 8247 mini_postgres.cc:96] Postgres bound to 47585
2026-09-25 14:59:49.380 UTC [20061] FATAL: database "slave" does not exist
127.8.13.212:47585 - accepting connections
I20260925 14:59:49.381016 8247 mini_ranger.cc:161] Starting Ranger...
I20260925 14:59:49.394932 8247 mini_ranger.cc:84] Created miniranger Postgres user
I20260925 14:59:49.832074 8247 mini_ranger.cc:87] Created ranger Postgres database
I20260925 14:59:49.832139 8247 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin
2026-09-25 14:59:50,102 [I] DB FLAVOR :POSTGRES
2026-09-25 14:59:50,103 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 14:59:50,103 [I] Checking connection..
2026-09-25 14:59:50,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-25 14:59:50,350 [I] Checking connection passed.
2026-09-25 14:59:50,350 [I] --------- Verifying version history table ---------
2026-09-25 14:59:50,350 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:50,600 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:50,600 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:50,870 [I] Table x_db_version_h does not exist in database ranger
2026-09-25 14:59:50,870 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-25 14:59:50,870 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-25 14:59:51.118 UTC [20125] WARNING: there is no transaction in progress
2026-09-25 14:59:51,124 [I] create_dbversion_catalog.sql file imported successfully
2026-09-25 14:59:51,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-25 14:59:51,371 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-25 14:59:51,372 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 14:59:51,372 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-25 14:59:51,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-25 14:59:51,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-15hz.c.gcp-upstream.internal','N') ;"
2026-09-25 14:59:52,094 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:52,346 [I] Table x_portal_user does not exist in database ranger
2026-09-25 14:59:52,346 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:52,580 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-25 14:59:52,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:52,823 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:53,054 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-25 14:59:53,054 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-25 14:59:54.013 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.030 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.046 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.062 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.078 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.094 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.149 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.156 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.171 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.181 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.191 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.204 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.214 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.220 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.227 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54.233 UTC [20233] WARNING: there is no transaction in progress
2026-09-25 14:59:54,878 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-25 14:59:54,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-25 14:59:55,107 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-25 14:59:55,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-25 14:59:55,349 [I] Table x_portal_user already exists in database 'ranger'
2026-09-25 14:59:55,349 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-25 14:59:55,603 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-25 14:59:55,603 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-25 14:59:55,844 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-25 14:59:56,072 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-15hz.c.gcp-upstream.internal';"
2026-09-25 14:59:56,306 [I] CORE_DB_SCHEMA import status has been updated
2026-09-25 14:59:56,306 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-25 14:59:56,553 [I] DB_PATCHES have already been applied
I20260925 14:59:56.558475 8247 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskxyjGoh/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260925 14:59:56.558532 8247 mini_ranger.cc:193] Using host: 127.8.13.212
I20260925 14:59:56.561437 8247 mini_ranger.cc:227] Ranger admin URL: http://127.8.13.212:54457
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 2:59:56 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 2:59:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp
Sep 25, 2026 2:59:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 25, 2026 2:59:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 25, 2026 2:59:57 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-admin/ews/webapp].
Sep 25, 2026 2:59:57 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260925 14:59:58.625105 8247 mini_ranger.cc:160] Time spent starting Ranger: real 9.244s user 0.002s sys 0.008s
Sep 25, 2026 3:00:17 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 25, 2026 3:00:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 25, 2026 3:00:17 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 25, 2026 3:00:17 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 25, 2026 3:00:18 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260925 15:00:19.315047 8247 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.13.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.13.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangerkms_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.13.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/rangerkms_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.13.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.13.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.13.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/HTTP_127.8.13.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 25 15:00:19 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260925 15:00:19.397675 8247 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260925 15:00:19.407112 8247 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260925 15:00:20.315863 8247 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260925 15:00:20.315950 8247 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms
I20260925 15:00:20.315966 8247 mini_ranger_kms.cc:227] Using postgres at 127.8.13.212:47585
2026-09-25 15:00:20,465 [I] DB FLAVOR :POSTGRES
2026-09-25 15:00:20,465 [I] --------- Verifying Ranger DB connection ---------
2026-09-25 15:00:20,465 [I] Checking connection
2026-09-25 15:00:20,465 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-25 15:00:20,708 [I] connection success
2026-09-25 15:00:20,708 [I] --------- Verifying Ranger DB tables ---------
2026-09-25 15:00:20,708 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-25 15:00:20,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-25 15:00:20,966 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-25 15:00:20,966 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-25 15:00:20,966 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-25 15:00:20,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.13.212:47585/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-25 15:00:21,230 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260925 15:00:21.705986 8247 mini_ranger_kms.cc:310] Created kms service
I20260925 15:00:21.927577 8247 mini_ranger_kms.cc:326] Created kudu user
I20260925 15:00:21.972774 8247 mini_ranger_kms.cc:343] Created rangerkms user
Sep 25 15:00:21 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348419, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:22.398792 8247 mini_ranger_kms.cc:384] Added ranger policy
I20260925 15:00:22.398912 8247 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms:/tmp/dist-test-taskxyjGoh/build/release/bin/postgresql.jar:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskxyjGoh/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260925 15:00:22.398929 8247 mini_ranger_kms.cc:240] Using host: 127.8.13.212
I20260925 15:00:22.402626 8247 mini_ranger_kms.cc:276] Ranger KMS PID: 20414
I20260925 15:00:22.402704 8247 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.13.212:37379
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-kms/ews/webapp].
Sep 25, 2026 3:00:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260925 15:00:24.473317 8247 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.076s user 0.001s sys 0.130s
I20260925 15:00:24.473393 8247 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260925 15:00:24.473436 8247 mini_ranger_kms.cc:401] 127.8.13.212:37379/kms/v1/keys
Sep 25 15:00:24 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348424, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 25 15:00:25 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348424, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
Sep 25 15:00:25 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348424, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.13.212@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
Sep 25 15:00:27 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:00:27.766646 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40789
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:40789
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:27.883504 20465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:27.883771 20465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:27.883795 20465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:27.885051 20465 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:27.885104 20465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:27.885118 20465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:27.885128 20465 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:27.885140 20465 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:27.886783 20465 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.13.212:37379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:40789
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:27.887020 20465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:27.887249 20465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:27.892345 20470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:27.892419 20471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:27.892540 20465 server_base.cc:1034] running on GCE node
W20260925 15:00:27.892345 20473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:27.892848 20465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:27.893180 20465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:27.894328 20465 hybrid_clock.cc:648] HybridClock initialized: now 1790348427894315 us; error 33 us; skew 500 ppm
Sep 25 15:00:27 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:27.896929 20465 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:27.897405 20465 webserver.cc:460] Webserver started at http://127.8.13.254:34305/ using document root <none> and password file <none>
I20260925 15:00:27.897631 20465 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:27.897671 20465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:27.897809 20465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:27 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:27.961457 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "9d5a13faf34243abab647743a7a5fec6"
format_stamp: "Formatted at 2026-09-25 15:00:27 on dist-test-slave-15hz"
server_key: "3bf49e670f68cf58a5e82772c81b8778"
server_key_iv: "20a2fb39cc86c2a1bae67445c2294e1e"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:27.961896 20465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "9d5a13faf34243abab647743a7a5fec6"
format_stamp: "Formatted at 2026-09-25 15:00:27 on dist-test-slave-15hz"
server_key: "3bf49e670f68cf58a5e82772c81b8778"
server_key_iv: "20a2fb39cc86c2a1bae67445c2294e1e"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:27.963196 20465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:00:27.977807 20480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:27.978036 20465 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:27.978157 20465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "9d5a13faf34243abab647743a7a5fec6"
format_stamp: "Formatted at 2026-09-25 15:00:27 on dist-test-slave-15hz"
server_key: "3bf49e670f68cf58a5e82772c81b8778"
server_key_iv: "20a2fb39cc86c2a1bae67445c2294e1e"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:27.978255 20465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:28.000173 20465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:28.002400 20465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:28.002552 20465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:28.007941 20532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:40789 every 8 connection(s)
I20260925 15:00:28.007941 20465 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:40789
I20260925 15:00:28.008529 20465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:28.009254 20533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:28.011608 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Bootstrap starting.
I20260925 15:00:28.012218 20533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:28.012487 20533 log.cc:824] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:28.013190 20533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: No bootstrap required, opened a new log
I20260925 15:00:28.014833 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 20465
I20260925 15:00:28.014928 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:00:28.015117 8247 external_mini_cluster.cc:1428] Setting key 11deb44d2542e5728fc20d58e231ad52
I20260925 15:00:28.015367 20533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:28.015688 20533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:28.015825 20533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5a13faf34243abab647743a7a5fec6, State: Initialized, Role: FOLLOWER
I20260925 15:00:28.016001 20533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:28.016083 20533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:28.016115 20533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:28.016175 20533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:28.016974 20533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:28.017086 20533 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d5a13faf34243abab647743a7a5fec6; no voters:
I20260925 15:00:28.017249 20533 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:00:28.017295 20538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:28.017470 20538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 LEADER]: Becoming Leader. State: Replica: 9d5a13faf34243abab647743a7a5fec6, State: Running, Role: LEADER
I20260925 15:00:28.017580 20533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:28.017592 20538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:28.017939 20538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d5a13faf34243abab647743a7a5fec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } } }
I20260925 15:00:28.018000 20538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:28.018097 20538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d5a13faf34243abab647743a7a5fec6. Latest consensus state: current_term: 1 leader_uuid: "9d5a13faf34243abab647743a7a5fec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } } }
I20260925 15:00:28.018138 20538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:28.018394 20543 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:28.018910 20543 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:28.019522 20533 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:00:28.020561 20543 catalog_manager.cc:1347] Generated new cluster ID: e95f4a1030a14ccc880431078d256243
I20260925 15:00:28.020606 20543 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:00:28 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348427, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:28.027423 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.015688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.8.13.254:40789)
0925 15:00:28.015897 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:28.015908 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:28.015946 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:00:28.016083 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:28.016091 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:28.016150 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:28.016301 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:28.017023 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.018431 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.019482 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.020127 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.021887 (+ 1760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.023025 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:28.023030 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:28.023030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:28.023043 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:28.025207 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.025646 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.025650 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.025650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.025784 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.026005 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.026006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.026006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.026361 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:28.026965 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:28.027042 (+ 77us) server_negotiation.cc:299] Negotiation successful
0925 15:00:28.027189 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":105,"threads_started":1}
I20260925 15:00:28.039019 20543 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:28.039958 20543 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:28.043069 20543 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Generated new TSK 0
I20260925 15:00:28.043365 20543 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:00:28.755600 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:28.842216 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:28.842418 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:28.842449 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:28.843631 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:28.843672 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:28.843706 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:28.845166 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:28.845387 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:28.845594 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:28.850311 20584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:28.850284 20586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:28.850291 20583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:28.850490 20578 server_base.cc:1034] running on GCE node
I20260925 15:00:28.850737 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:28.851009 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:28.852228 20578 hybrid_clock.cc:648] HybridClock initialized: now 1790348428852206 us; error 34 us; skew 500 ppm
Sep 25 15:00:28 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:28.855445 20578 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:28.856082 20578 webserver.cc:460] Webserver started at http://127.8.13.193:36245/ using document root <none> and password file <none>
I20260925 15:00:28.856351 20578 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:28.856429 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:28.856581 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:28 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:28.869186 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "2a58aba2e7b74461a9ef98ee9e642d0c"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-15hz"
server_key: "c39c8634e2aa5159671fc5c676a003dc"
server_key_iv: "2f3606398e082972bea59354a4ddf750"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.869691 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "2a58aba2e7b74461a9ef98ee9e642d0c"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-15hz"
server_key: "c39c8634e2aa5159671fc5c676a003dc"
server_key_iv: "2f3606398e082972bea59354a4ddf750"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.871119 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:28.887763 20593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.888029 20578 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260925 15:00:28.888121 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "2a58aba2e7b74461a9ef98ee9e642d0c"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-15hz"
server_key: "c39c8634e2aa5159671fc5c676a003dc"
server_key_iv: "2f3606398e082972bea59354a4ddf750"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:28.888214 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:28.901890 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:28.904309 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:28.904464 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:28.904727 20578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:28.905064 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:28.905104 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.905189 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:28.905216 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:28.912595 20578 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:33491
I20260925 15:00:28.912817 20706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:33491 every 8 connection(s)
I20260925 15:00:28.913087 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
Sep 25 15:00:28 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:28.922981 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 20578
I20260925 15:00:28.923076 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:00:28.923286 8247 external_mini_cluster.cc:1428] Setting key e9b6ac1ec8807b734d35efec5c8a29f6
I20260925 15:00:28.925459 20709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.913863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.193:57019)
0925 15:00:28.914040 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:28.914100 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:28.914736 (+ 636us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:28.915163 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:28.915168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:28.915415 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:28.915758 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.915765 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.916805 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.916806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.918146 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:28.918149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.918681 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.918682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:28.918825 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.919479 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:28.919489 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:28.921187 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:28.923263 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:28.923272 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:28.923275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:28.924089 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:28.924323 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:28.924324 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:28.924325 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:28.924462 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:28.924774 (+ 312us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:28.924780 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:28.924955 (+ 175us) client_negotiation.cc:769] Sending connection context
0925 15:00:28.925049 (+ 94us) client_negotiation.cc:240] Negotiation successful
0925 15:00:28.925210 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":81,"threads_started":1}
I20260925 15:00:28.925912 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:28.914097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:57019 (local address 127.8.13.254:40789)
0925 15:00:28.914932 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:28.914934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:28.914941 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:28.914966 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:28.914966 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:28.914979 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:28.915044 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:28.915865 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.916696 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.918244 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:28.918528 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:28.918607 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:28.921316 (+ 2709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:28.921318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:28.921318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:28.921326 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:28.923163 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.924174 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.924175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.924175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.924243 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:28.924527 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:28.924528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:28.924528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:28.924689 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:28.924747 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:28.925717 (+ 970us) server_negotiation.cc:299] Negotiation successful
0925 15:00:28.925814 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":802,"thread_start_us":69,"threads_started":1}
I20260925 15:00:28.926491 20707 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:28.926662 20707 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:28.926920 20707 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:28.927435 20497 ts_manager.cc:194] Registered new tserver with Master: 2a58aba2e7b74461a9ef98ee9e642d0c (127.8.13.193:33491)
I20260925 15:00:28.928105 20497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:57019
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:00:28.954021 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:29.069378 20715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:29.069585 20715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:29.069626 20715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:29.070818 20715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:29.070861 20715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:29.070894 20715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:29.072402 20715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:29.072643 20715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:29.072892 20715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:29.077672 20722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.077840 20725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.077672 20723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.078624 20715 server_base.cc:1034] running on GCE node
I20260925 15:00:29.078835 20715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:29.079077 20715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:29.080225 20715 hybrid_clock.cc:648] HybridClock initialized: now 1790348429080211 us; error 25 us; skew 500 ppm
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.082742 20715 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:29.083235 20715 webserver.cc:460] Webserver started at http://127.8.13.194:43777/ using document root <none> and password file <none>
I20260925 15:00:29.083467 20715 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:29.083513 20715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:29.083686 20715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:29.103197 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "a82ba8f914e34dcd9d074c968e57b834"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "2a4b0ebdb126d470858d80dabc2856e0"
server_key_iv: "54177eb092dc85be00cf2ab7f342a841"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.103703 20715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "a82ba8f914e34dcd9d074c968e57b834"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "2a4b0ebdb126d470858d80dabc2856e0"
server_key_iv: "54177eb092dc85be00cf2ab7f342a841"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.106617 20715 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.127156 20732 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.139904 20715 fs_manager.cc:725] Time spent opening block manager: real 0.018s user 0.001s sys 0.000s
I20260925 15:00:29.140034 20715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "a82ba8f914e34dcd9d074c968e57b834"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "2a4b0ebdb126d470858d80dabc2856e0"
server_key_iv: "54177eb092dc85be00cf2ab7f342a841"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.140131 20715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:29.166054 20715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:29.168663 20715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:29.168896 20715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:29.169202 20715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:29.170929 20715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:29.171036 20715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.171113 20715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:29.171180 20715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.177312 20715 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:41217
I20260925 15:00:29.177561 20845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:41217 every 8 connection(s)
I20260925 15:00:29.178859 20715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:29.184541 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 20715
I20260925 15:00:29.184648 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:00:29.184860 8247 external_mini_cluster.cc:1428] Setting key 006124979b0cfe5aafa7aaf096027cca
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:29.189302 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.180478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:49377 (local address 127.8.13.254:40789)
0925 15:00:29.180541 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.180543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.180771 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:00:29.181273 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.181274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.181291 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.181340 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.181884 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.182391 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.183406 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.183747 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.183816 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.186125 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.186126 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.186126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.186135 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.187697 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.188228 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.188229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.188229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.188292 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.188562 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.188563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.188563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.188692 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.188738 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.189118 (+ 380us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.189209 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:00:29.189843 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.180451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.194:49377)
0925 15:00:29.180629 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.180679 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.181174 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.181411 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.181414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.181600 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:29.181794 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.181798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.182469 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.182469 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.183324 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.183326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.183838 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.183839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.183957 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.184488 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:29.184500 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:29.186018 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:29.187794 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.187805 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.187808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.188135 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.188360 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.188361 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.188361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.188498 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.188793 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:29.188800 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:29.188973 (+ 173us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.189441 (+ 468us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.189606 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":105,"threads_started":1}
I20260925 15:00:29.190446 20846 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:29.190582 20846 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.190821 20846 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:29.191424 20497 ts_manager.cc:194] Registered new tserver with Master: a82ba8f914e34dcd9d074c968e57b834 (127.8.13.194:41217)
I20260925 15:00:29.191956 20497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:49377
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:00:29.212692 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:29.337212 20852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:29.337563 20852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:29.337661 20852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:29.339474 20852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:29.339751 20852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:29.339861 20852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:29.342248 20852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:29.342736 20852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:29.343066 20852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:29.349489 20858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.349884 20857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.350893 20852 server_base.cc:1034] running on GCE node
W20260925 15:00:29.351073 20860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.351339 20852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:29.351661 20852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:29.352828 20852 hybrid_clock.cc:648] HybridClock initialized: now 1790348429352808 us; error 33 us; skew 500 ppm
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.355968 20852 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:29.356472 20852 webserver.cc:460] Webserver started at http://127.8.13.195:36281/ using document root <none> and password file <none>
I20260925 15:00:29.356737 20852 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:29.356787 20852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:29.356909 20852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:29.383827 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "ff6fca8aa31f42feb634a94b9bee7ac4"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "6fb88c32876bace4948ac9ce21a8c1b9"
server_key_iv: "fbcf7b08a85d587463c8c6ec608b89b9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.384420 20852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "ff6fca8aa31f42feb634a94b9bee7ac4"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "6fb88c32876bace4948ac9ce21a8c1b9"
server_key_iv: "fbcf7b08a85d587463c8c6ec608b89b9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.390535 20852 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.000s
I20260925 15:00:29.408180 20867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.408459 20852 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.000s
I20260925 15:00:29.408555 20852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "ff6fca8aa31f42feb634a94b9bee7ac4"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "6fb88c32876bace4948ac9ce21a8c1b9"
server_key_iv: "fbcf7b08a85d587463c8c6ec608b89b9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.408667 20852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:29.441206 20852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:29.443501 20852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:29.443740 20852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:29.443987 20852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:29.444384 20852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:29.444455 20852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.444509 20852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:29.444543 20852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.451267 20852 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:36345
I20260925 15:00:29.451792 20852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:29.451786 20980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:36345 every 8 connection(s)
I20260925 15:00:29.459810 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 20852
I20260925 15:00:29.459920 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:00:29.460106 8247 external_mini_cluster.cc:1428] Setting key 4592a618ad4186cebea0e3e40b82eb93
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:29.472234 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.452925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:41673 (local address 127.8.13.254:40789)
0925 15:00:29.452976 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.452978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.452990 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:00:29.453745 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.453746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.453759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.453824 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.454645 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.455380 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.465031 (+ 9651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.465329 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.465426 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.468922 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.468924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.468924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.468932 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.470539 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.471081 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.471083 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.471083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.471146 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.471430 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.471430 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.471431 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.471593 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.471648 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.472032 (+ 384us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.472129 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:00:29.473043 20983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.452659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.195:41673)
0925 15:00:29.452848 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.452905 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.453610 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.453916 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.453921 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.454210 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:29.454535 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.454543 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.455486 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.455487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.464887 (+ 9400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.464895 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.465475 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.465477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.465741 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.466395 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:29.466405 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:29.468809 (+ 2404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:29.470653 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.470662 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.470665 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.470989 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.471215 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.471216 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.471216 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.471358 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.471694 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:29.471700 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:29.471880 (+ 180us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.472600 (+ 720us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.472793 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260925 15:00:29.473758 20981 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:29.473857 20981 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.474049 20981 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:29.474472 20497 ts_manager.cc:194] Registered new tserver with Master: ff6fca8aa31f42feb634a94b9bee7ac4 (127.8.13.195:36345)
I20260925 15:00:29.474983 20497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:41673
I20260925 15:00:29.477095 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:29.477779 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 20578
I20260925 15:00:29.484861 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 20715
I20260925 15:00:29.490585 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 20852
I20260925 15:00:29.496135 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 20465
I20260925 15:00:29.501022 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40789
--webserver_interface=127.8.13.254
--webserver_port=34305
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:40789
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:29.583559 20984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:29.583810 20984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:29.583835 20984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:29.585110 20984 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:29.585155 20984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:29.585167 20984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:29.585177 20984 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:29.585188 20984 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:29.586805 20984 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:40789
--ranger_config_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:29.587049 20984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:29.587278 20984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:29.592108 20992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.592109 20990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.592109 20989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.592617 20984 server_base.cc:1034] running on GCE node
I20260925 15:00:29.592880 20984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:29.593197 20984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:29.594390 20984 hybrid_clock.cc:648] HybridClock initialized: now 1790348429594371 us; error 27 us; skew 500 ppm
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.597084 20984 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:29.597574 20984 webserver.cc:460] Webserver started at http://127.8.13.254:34305/ using document root <none> and password file <none>
I20260925 15:00:29.597807 20984 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:29.597862 20984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:29.656651 20984 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:29.657330 21000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.657502 20984 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:29.657585 20984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "9d5a13faf34243abab647743a7a5fec6"
format_stamp: "Formatted at 2026-09-25 15:00:27 on dist-test-slave-15hz"
server_key: "3bf49e670f68cf58a5e82772c81b8778"
server_key_iv: "20a2fb39cc86c2a1bae67445c2294e1e"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.657960 20984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:29.703130 20984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:29.705118 20984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:29.705271 20984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:29.709091 20984 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:40789
I20260925 15:00:29.709138 21052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:40789 every 8 connection(s)
I20260925 15:00:29.709494 20984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:29.710793 21053 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:00:29.711460 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Bootstrap starting.
I20260925 15:00:29.718261 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 20984
I20260925 15:00:29.718622 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:33491
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:40789
--webserver_port=36245
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:29.719472 21053 log.cc:824] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:29.721349 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:29.721618 21053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Bootstrap complete.
I20260925 15:00:29.723965 21053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:29.724203 21053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d5a13faf34243abab647743a7a5fec6, State: Initialized, Role: FOLLOWER
I20260925 15:00:29.724372 21053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:29.724444 21053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:29.724478 21053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:29.724524 21053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:00:29.725638 21053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:29.725730 21053 leader_election.cc:302] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d5a13faf34243abab647743a7a5fec6; no voters:
I20260925 15:00:29.725893 21053 leader_election.cc:288] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:00:29.725940 21057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:00:29.726078 21057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [term 2 LEADER]: Becoming Leader. State: Replica: 9d5a13faf34243abab647743a7a5fec6, State: Running, Role: LEADER
I20260925 15:00:29.726166 21053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:29.726500 21057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } }
I20260925 15:00:29.726792 21059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d5a13faf34243abab647743a7a5fec6. Latest consensus state: current_term: 2 leader_uuid: "9d5a13faf34243abab647743a7a5fec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } } }
I20260925 15:00:29.726871 21059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:29.727001 21058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9d5a13faf34243abab647743a7a5fec6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d5a13faf34243abab647743a7a5fec6" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40789 } } }
I20260925 15:00:29.727233 21058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:29.727208 21066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:29.727885 21066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:29.728008 21066 catalog_manager.cc:1259] Loaded cluster ID: e95f4a1030a14ccc880431078d256243
I20260925 15:00:29.728040 21066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:00:29.728317 21053 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260925 15:00:29.729168 21066 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:29.729355 21066 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Loaded TSK: 0
I20260925 15:00:29.729570 21066 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:00:29.809165 21056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:29.809382 21056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:29.809417 21056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:29.810824 21056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:29.810881 21056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:29.810920 21056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:29.812573 21056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:33491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=36245
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:29.812856 21056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:29.813133 21056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:29.818486 21093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.818640 21091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:29.818746 21090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:29.818922 21056 server_base.cc:1034] running on GCE node
I20260925 15:00:29.819128 21056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:29.819373 21056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:29.820582 21056 hybrid_clock.cc:648] HybridClock initialized: now 1790348429820540 us; error 28 us; skew 500 ppm
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:29.823997 21056 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:29.824513 21056 webserver.cc:460] Webserver started at http://127.8.13.193:36245/ using document root <none> and password file <none>
I20260925 15:00:29.824769 21056 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:29.824836 21056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:29.855988 21056 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260925 15:00:29.857105 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.857283 21056 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:29.857399 21056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "2a58aba2e7b74461a9ef98ee9e642d0c"
format_stamp: "Formatted at 2026-09-25 15:00:28 on dist-test-slave-15hz"
server_key: "c39c8634e2aa5159671fc5c676a003dc"
server_key_iv: "2f3606398e082972bea59354a4ddf750"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:29.857782 21056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:29.871703 21056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:29.874070 21056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:29.874228 21056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:29.874537 21056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:29.874990 21056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:29.875093 21056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.875201 21056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:29.875279 21056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:29.881776 21056 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:33491
I20260925 15:00:29.882200 21056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:29.884375 21214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:33491 every 8 connection(s)
I20260925 15:00:29.888092 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21056
I20260925 15:00:29.888404 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:41217
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:40789
--webserver_port=43777
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:00:29 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:29.896536 21217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.882976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.193:43233)
0925 15:00:29.883729 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:29.883774 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:29.884300 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:29.884800 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:29.884804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:29.885003 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:29.885203 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.885208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.886381 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.886382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.887195 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:29.887199 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.888204 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.888205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:29.888353 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.888906 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:29.888917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:29.892121 (+ 3204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:29.894578 (+ 2457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.894585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.894587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.894887 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.895151 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:29.895152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:29.895153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:29.895283 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:29.895821 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:29.895829 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:29.895996 (+ 167us) client_negotiation.cc:769] Sending connection context
0925 15:00:29.896107 (+ 111us) client_negotiation.cc:240] Negotiation successful
0925 15:00:29.896282 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":75,"threads_started":1}
I20260925 15:00:29.903723 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:29.883403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43233 (local address 127.8.13.254:40789)
0925 15:00:29.883651 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:29.883663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:29.883892 (+ 229us) server_negotiation.cc:407] Connection header received
0925 15:00:29.884535 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:29.884542 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:29.884605 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:29.884730 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:29.885325 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.886249 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.887293 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:29.887960 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:29.888069 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:29.892560 (+ 4491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:29.892566 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:29.892566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:29.892581 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:29.894475 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.894975 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.894979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.894979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.895088 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:29.895352 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:29.895353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:29.895353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:29.895700 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:29.895791 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:29.903229 (+ 7438us) server_negotiation.cc:299] Negotiation successful
0925 15:00:29.903413 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":81,"threads_started":1}
I20260925 15:00:29.904866 21215 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:29.904968 21215 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.905167 21215 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:29.906025 21215 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:40789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:00:29.906091 21215 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.906149 21215 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:29.906530 21215 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:29.906597 21215 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:29.906927 21215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:00:30.009541 21219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:30.009836 21219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:30.009878 21219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:30.011809 21219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:30.011925 21219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:30.012002 21219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:30.014384 21219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:41217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=43777
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:30.014885 21219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:30.015201 21219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:30.020695 21224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:30.020900 21219 server_base.cc:1034] running on GCE node
W20260925 15:00:30.020895 21225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:30.021122 21227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:30.021363 21219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:30.021667 21219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:30.022877 21219 hybrid_clock.cc:648] HybridClock initialized: now 1790348430022852 us; error 37 us; skew 500 ppm
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:30.025938 21219 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:30.026600 21219 webserver.cc:460] Webserver started at http://127.8.13.194:43777/ using document root <none> and password file <none>
I20260925 15:00:30.026947 21219 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:30.027030 21219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:30.045512 21219 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260925 15:00:30.046465 21235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.046744 21219 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:30.046846 21219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "a82ba8f914e34dcd9d074c968e57b834"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "2a4b0ebdb126d470858d80dabc2856e0"
server_key_iv: "54177eb092dc85be00cf2ab7f342a841"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:30.047217 21219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:30.064114 21219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:30.066498 21219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:30.066625 21219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:30.066852 21219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:30.067162 21219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:30.067196 21219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.067219 21219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:30.067229 21219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.075927 21219 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:41217
I20260925 15:00:30.076349 21219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:30.079008 21348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:41217 every 8 connection(s)
I20260925 15:00:30.081883 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21219
I20260925 15:00:30.082170 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.13.212:37379/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:36345
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:40789
--webserver_port=36281
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:30.095100 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.079766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:59059 (local address 127.8.13.254:40789)
0925 15:00:30.079813 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:30.079815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:30.080109 (+ 294us) server_negotiation.cc:407] Connection header received
0925 15:00:30.080621 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:30.080623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:30.080640 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:30.080684 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:30.081424 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.082267 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.083843 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.084138 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.084207 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.091397 (+ 7190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:30.091399 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:30.091399 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:30.091409 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:30.093318 (+ 1909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.093904 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.093906 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.093906 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.093974 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.094301 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.094302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.094302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.094445 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:30.094487 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:30.094893 (+ 406us) server_negotiation.cc:299] Negotiation successful
0925 15:00:30.094988 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:00:30.095734 21351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.079801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.194:59059)
0925 15:00:30.079970 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:30.080014 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:30.080509 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:30.080762 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:30.080767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:30.081012 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:30.081304 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:30.081310 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.082388 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.082389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:30.083745 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:30.083749 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.084316 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.084317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:30.084463 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.089455 (+ 4992us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:30.089471 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:30.091236 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:30.093432 (+ 2196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:30.093445 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:30.093448 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:30.093782 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:30.094073 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:30.094074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:30.094075 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:30.094229 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:30.094532 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:30.094540 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:30.094728 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 15:00:30.095257 (+ 529us) client_negotiation.cc:240] Negotiation successful
0925 15:00:30.095441 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260925 15:00:30.096201 21349 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:30.096309 21349 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.096518 21349 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:30.097116 21349 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:40789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:00:30.097219 21349 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.097295 21349 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:30.097579 21349 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.097658 21349 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:30.097934 21349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:00:30.188464 21352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:30.188777 21352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:30.188818 21352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:30.190814 21352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:30.190886 21352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:30.190928 21352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:30.193321 21352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.13.212:37379/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:36345
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=36281
--tserver_master_addrs=127.8.13.254:40789
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:30.193606 21352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:30.193912 21352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:30.200423 21357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:30.200428 21358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:30.201545 21352 server_base.cc:1034] running on GCE node
W20260925 15:00:30.202066 21360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:30.202322 21352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:30.202615 21352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:30.205894 21352 hybrid_clock.cc:648] HybridClock initialized: now 1790348430205858 us; error 45 us; skew 500 ppm
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:30.209110 21352 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:30.209692 21352 webserver.cc:460] Webserver started at http://127.8.13.195:36281/ using document root <none> and password file <none>
I20260925 15:00:30.210026 21352 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:30.210153 21352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:30.224709 21352 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:00:30.232033 21368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.233893 21352 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.000s sys 0.000s
I20260925 15:00:30.234090 21352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "ff6fca8aa31f42feb634a94b9bee7ac4"
format_stamp: "Formatted at 2026-09-25 15:00:29 on dist-test-slave-15hz"
server_key: "6fb88c32876bace4948ac9ce21a8c1b9"
server_key_iv: "fbcf7b08a85d587463c8c6ec608b89b9"
server_key_version: "kuduclusterkey@0"
I20260925 15:00:30.234732 21352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:30.262720 21352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:30.265164 21352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:30.265347 21352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:30.265637 21352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:30.266039 21352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:30.266093 21352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.266124 21352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:30.266141 21352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:30.273041 21352 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:36345
I20260925 15:00:30.273126 21482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:36345 every 8 connection(s)
I20260925 15:00:30.273451 21352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:30.274375 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21352
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:30.286125 21487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.274744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39596 (local address 127.8.13.254:40789)
0925 15:00:30.274859 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:30.274860 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:30.275043 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:00:30.275110 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:30.275111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:30.275124 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:30.275196 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:30.277267 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.278051 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.279452 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.279799 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.279870 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.281697 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:30.281699 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:30.281699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:30.281709 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:30.284696 (+ 2987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.285244 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.285245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.285246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.285308 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.285525 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.285526 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.285526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.285676 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:30.285844 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:30.285929 (+ 85us) server_negotiation.cc:299] Negotiation successful
0925 15:00:30.286014 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
I20260925 15:00:30.287230 21218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.274174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:50131 (local address 127.8.13.254:40789)
0925 15:00:30.274252 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:30.274255 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:30.274748 (+ 493us) server_negotiation.cc:407] Connection header received
0925 15:00:30.275373 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:30.275373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:30.275383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:30.275417 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:30.276129 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.276881 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.278405 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.278715 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.278779 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.282268 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:30.282270 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:30.282271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:30.282278 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:30.283938 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.285401 (+ 1463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.285402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.285402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.285458 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:30.286237 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:30.286238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:30.286238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:30.286365 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:30.286404 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:30.287055 (+ 651us) server_negotiation.cc:299] Negotiation successful
0925 15:00:30.287146 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260925 15:00:30.287741 21485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:30.274149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40789 (local address 127.8.13.195:50131)
0925 15:00:30.274608 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:30.274659 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:30.275286 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:30.275501 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:30.275506 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:30.275745 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:30.276028 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:30.276035 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.276999 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.277001 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:30.278317 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:30.278321 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:30.278872 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:30.278874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:30.279027 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:30.280004 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:30.280015 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:30.282110 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:30.284806 (+ 2696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:30.284818 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:30.284820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:30.285161 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:30.285902 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:30.285903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:30.285904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:30.286064 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:30.286620 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:30.286629 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:30.286885 (+ 256us) client_negotiation.cc:769] Sending connection context
0925 15:00:30.287312 (+ 427us) client_negotiation.cc:240] Negotiation successful
0925 15:00:30.287485 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":78,"threads_started":1}
I20260925 15:00:30.288187 21483 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40789
I20260925 15:00:30.288308 21483 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.288519 21483 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:30.289227 21483 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:40789 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260925 15:00:30.289305 21483 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.289373 21483 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:30.289649 21483 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.289734 21483 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
W20260925 15:00:30.289991 21483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 25 15:00:30 dist-test-slave-15hz krb5kdc[20020](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:30.907151 21215 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:30.907322 21215 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:30.907797 21017 ts_manager.cc:194] Registered new tserver with Master: 2a58aba2e7b74461a9ef98ee9e642d0c (127.8.13.193:33491)
I20260925 15:00:30.908792 21017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:43233
I20260925 15:00:31.098126 21349 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:31.098299 21349 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:31.098639 21016 ts_manager.cc:194] Registered new tserver with Master: a82ba8f914e34dcd9d074c968e57b834 (127.8.13.194:41217)
I20260925 15:00:31.099191 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:59059
Sep 25 15:00:31 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:31.290203 21483 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:31.290381 21483 heartbeater.cc:511] Master 127.8.13.254:40789 requested a full tablet report, sending...
I20260925 15:00:31.290781 21016 ts_manager.cc:194] Registered new tserver with Master: ff6fca8aa31f42feb634a94b9bee7ac4 (127.8.13.195:36345)
I20260925 15:00:31.291177 21016 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:50131
I20260925 15:00:31.295310 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:31.304770 21505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.296371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39606 (local address 127.8.13.254:40789)
0925 15:00:31.296637 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.296640 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.296647 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:31.296673 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.296674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.296686 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.296756 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:31.297508 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.298286 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.299376 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.299721 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.299788 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.301380 (+ 1592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:31.301381 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:31.301382 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:31.301390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:31.303262 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:31.303749 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:31.303751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:31.303752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:31.303821 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:31.304031 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:31.304032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:31.304032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:31.304184 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:31.304300 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.304436 (+ 136us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.304528 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":66,"threads_started":1}
I20260925 15:00:31.307153 21016 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:00:31.309598 21016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:00:31.322592 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.317448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:36345 (local address 127.0.0.1:58588)
0925 15:00:31.317782 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.317825 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.317907 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.318296 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.318300 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.318302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.318376 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.318378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.319735 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.319736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.321089 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.321092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.322224 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.322225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.322324 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.322339 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.322412 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.322471 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":85,"threads_started":1}
I20260925 15:00:31.323328 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.317570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58588 (local address 127.8.13.195:36345)
0925 15:00:31.318054 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.318060 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.318070 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:00:31.318108 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.318110 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.318126 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.318203 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.318462 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.319628 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.321212 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.322036 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.322116 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.322184 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.323184 (+ 1000us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.323238 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":84,"threads_started":1}
I20260925 15:00:31.324313 21417 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c (DEFAULT_TABLE table=test-table [id=f82fe3dae5744387b27b528f39228a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:31.324719 21417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a39935a3c4c028d6b7872d5b25d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:31.325994 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.317716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33491 (local address 127.0.0.1:36998)
0925 15:00:31.318836 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.318846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.318898 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.319146 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.319147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.319149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.319190 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.319192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.320770 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.320771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.321901 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.321905 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.325764 (+ 3859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.325765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.325833 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.325838 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.325878 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.325921 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1103,"thread_start_us":67,"threads_started":1}
I20260925 15:00:31.326252 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.318144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36998 (local address 127.8.13.193:33491)
0925 15:00:31.318719 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.318723 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.318949 (+ 226us) server_negotiation.cc:407] Connection header received
0925 15:00:31.318983 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.318985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.319000 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.319065 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.319252 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.320686 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.322004 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.323136 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.323211 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.323265 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.326127 (+ 2862us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.326180 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":46,"threads_started":1}
I20260925 15:00:31.326915 21138 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c (DEFAULT_TABLE table=test-table [id=f82fe3dae5744387b27b528f39228a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:31.327272 21138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a39935a3c4c028d6b7872d5b25d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:31.328239 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.318270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35796 (local address 127.8.13.194:41217)
0925 15:00:31.318783 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.318786 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.323296 (+ 4510us) server_negotiation.cc:407] Connection header received
0925 15:00:31.323339 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.323342 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.323358 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.323427 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.323650 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.324646 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.327025 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.327802 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.327868 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.327923 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.328122 (+ 199us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.328165 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":44,"threads_started":1}
I20260925 15:00:31.328454 21510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.317860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41217 (local address 127.0.0.1:35796)
0925 15:00:31.323040 (+ 5180us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.323051 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.323096 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.323513 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.323514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.323516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.323571 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.323572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.324735 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.324735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.326895 (+ 2160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.326898 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.327957 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.327958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.328015 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.328018 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.328058 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.328384 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5165,"thread_start_us":73,"threads_started":1}
I20260925 15:00:31.329149 21283 tablet_service.cc:1467] Processing CreateTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c (DEFAULT_TABLE table=test-table [id=f82fe3dae5744387b27b528f39228a9e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:31.329321 21516 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Bootstrap starting.
I20260925 15:00:31.329476 21283 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac1a39935a3c4c028d6b7872d5b25d1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:31.330034 21516 tablet_bootstrap.cc:654] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:31.330350 21516 log.cc:824] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:31.331197 21518 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Bootstrap starting.
I20260925 15:00:31.332072 21518 tablet_bootstrap.cc:654] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:31.332451 21518 log.cc:824] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:31.333268 21518 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: No bootstrap required, opened a new log
I20260925 15:00:31.333333 21518 ts_tablet_manager.cc:1397] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:31.333707 21520 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Bootstrap starting.
I20260925 15:00:31.334354 21520 tablet_bootstrap.cc:654] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:31.334800 21520 log.cc:824] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:31.335777 21520 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: No bootstrap required, opened a new log
I20260925 15:00:31.335843 21520 ts_tablet_manager.cc:1397] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:00:31.335875 21518 raft_consensus.cc:348] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.336032 21518 raft_consensus.cc:374] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:31.336063 21518 raft_consensus.cc:729] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a58aba2e7b74461a9ef98ee9e642d0c, State: Initialized, Role: FOLLOWER
I20260925 15:00:31.336582 21518 consensus_queue.cc:260] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.336861 21518 ts_tablet_manager.cc:1428] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:00:31.336970 21215 heartbeater.cc:503] Master 127.8.13.254:40789 was elected leader, sending a full tablet report...
I20260925 15:00:31.338284 21520 raft_consensus.cc:348] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.338413 21520 raft_consensus.cc:374] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:31.338433 21520 raft_consensus.cc:729] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a82ba8f914e34dcd9d074c968e57b834, State: Initialized, Role: FOLLOWER
I20260925 15:00:31.338660 21520 consensus_queue.cc:260] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.338956 21349 heartbeater.cc:503] Master 127.8.13.254:40789 was elected leader, sending a full tablet report...
I20260925 15:00:31.338955 21520 ts_tablet_manager.cc:1428] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:00:31.339934 21516 tablet_bootstrap.cc:492] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: No bootstrap required, opened a new log
I20260925 15:00:31.340015 21516 ts_tablet_manager.cc:1397] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260925 15:00:31.342026 21516 raft_consensus.cc:348] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.342171 21516 raft_consensus.cc:374] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:31.342202 21516 raft_consensus.cc:729] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff6fca8aa31f42feb634a94b9bee7ac4, State: Initialized, Role: FOLLOWER
I20260925 15:00:31.342339 21516 consensus_queue.cc:260] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.342613 21483 heartbeater.cc:503] Master 127.8.13.254:40789 was elected leader, sending a full tablet report...
I20260925 15:00:31.342624 21516 ts_tablet_manager.cc:1428] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20260925 15:00:31.392719 21216 tablet.cc:2369] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 25 15:00:31 dist-test-slave-15hz krb5kdc[20020](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348430, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for HTTP/127.8.13.212@KRBTEST.COM
I20260925 15:00:31.523849 21524 raft_consensus.cc:482] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:31.523991 21524 raft_consensus.cc:504] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.524375 21524 leader_election.cc:288] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a58aba2e7b74461a9ef98ee9e642d0c (127.8.13.193:33491), a82ba8f914e34dcd9d074c968e57b834 (127.8.13.194:41217)
W20260925 15:00:31.524899 21484 tablet.cc:2369] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:00:31.528064 21522 raft_consensus.cc:482] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:31.528203 21522 raft_consensus.cc:504] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.528633 21522 leader_election.cc:288] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff6fca8aa31f42feb634a94b9bee7ac4 (127.8.13.195:36345), a82ba8f914e34dcd9d074c968e57b834 (127.8.13.194:41217)
I20260925 15:00:31.532045 21514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.525008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:40835 (local address 127.8.13.193:33491)
0925 15:00:31.525105 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.525107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.526803 (+ 1696us) server_negotiation.cc:407] Connection header received
0925 15:00:31.526915 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.526917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.526932 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.526965 (+ 33us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.527224 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.527935 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.531026 (+ 3091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.531623 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.531670 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.531682 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.531924 (+ 242us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.531972 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:00:31.532524 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.524543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33491 (local address 127.8.13.195:40835)
0925 15:00:31.526728 (+ 2185us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.526744 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.526837 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.527044 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.527045 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.527048 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.527136 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.527139 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.529682 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.529683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.530914 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.530919 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.531765 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.531766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.531836 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.531841 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.532366 (+ 525us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.532420 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2158,"thread_start_us":98,"threads_started":1}
I20260925 15:00:31.533139 21165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" is_pre_election: true
I20260925 15:00:31.533285 21165 raft_consensus.cc:2454] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6fca8aa31f42feb634a94b9bee7ac4 in term 0.
I20260925 15:00:31.533531 21369 leader_election.cc:302] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a58aba2e7b74461a9ef98ee9e642d0c, ff6fca8aa31f42feb634a94b9bee7ac4; no voters:
I20260925 15:00:31.533694 21524 raft_consensus.cc:2781] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:31.533746 21524 raft_consensus.cc:482] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:31.533768 21524 raft_consensus.cc:3037] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:31.534694 21524 raft_consensus.cc:504] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.534835 21524 leader_election.cc:288] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [CANDIDATE]: Term 1 election: Requested vote from peers 2a58aba2e7b74461a9ef98ee9e642d0c (127.8.13.193:33491), a82ba8f914e34dcd9d074c968e57b834 (127.8.13.194:41217)
I20260925 15:00:31.535163 21165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c"
I20260925 15:00:31.535244 21165 raft_consensus.cc:3037] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:31.536175 21165 raft_consensus.cc:2454] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6fca8aa31f42feb634a94b9bee7ac4 in term 1.
I20260925 15:00:31.536438 21369 leader_election.cc:302] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a58aba2e7b74461a9ef98ee9e642d0c, ff6fca8aa31f42feb634a94b9bee7ac4; no voters:
I20260925 15:00:31.536590 21524 raft_consensus.cc:2781] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:31.536746 21524 raft_consensus.cc:686] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 1 LEADER]: Becoming Leader. State: Replica: ff6fca8aa31f42feb634a94b9bee7ac4, State: Running, Role: LEADER
I20260925 15:00:31.536844 21524 consensus_queue.cc:237] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } }
I20260925 15:00:31.537815 21016 catalog_manager.cc:5495] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff6fca8aa31f42feb634a94b9bee7ac4 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 36345 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 } health_report { overall_health: UNKNOWN } } }
I20260925 15:00:31.539229 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.529345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:58497 (local address 127.8.13.195:36345)
0925 15:00:31.529404 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.529407 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.535858 (+ 6451us) server_negotiation.cc:407] Connection header received
0925 15:00:31.535955 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.535956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.535967 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.535993 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.536187 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.536887 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.538299 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.538821 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.538870 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.538879 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.539123 (+ 244us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.539166 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:00:31.539444 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.528896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:36345 (local address 127.8.13.193:58497)
0925 15:00:31.535786 (+ 6890us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.535797 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.535886 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.536051 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.536052 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.536054 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.536119 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.536121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.536990 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.536990 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.538191 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.538195 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.538956 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.538957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.539043 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.539047 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.539308 (+ 261us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.539354 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6858,"thread_start_us":106,"threads_started":1}
I20260925 15:00:31.539757 21437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" is_pre_election: true
I20260925 15:00:31.541452 21526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.524781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41217 (local address 127.8.13.195:40075)
0925 15:00:31.533011 (+ 8230us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.533022 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.533095 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.538179 (+ 5084us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.538180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.538182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.538233 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.538234 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.539142 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.539143 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.540498 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.540500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.541247 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.541247 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.541304 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.541308 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.541344 (+ 36us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.541389 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8216,"thread_start_us":62,"threads_started":1}
I20260925 15:00:31.541823 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.525057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:40075 (local address 127.8.13.194:41217)
0925 15:00:31.525139 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.525141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.533064 (+ 7923us) server_negotiation.cc:407] Connection header received
0925 15:00:31.533165 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.533166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.533181 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.533210 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.538331 (+ 5121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.538973 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.540620 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.541139 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.541207 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.541217 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.541713 (+ 496us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.541764 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260925 15:00:31.542316 21303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82ba8f914e34dcd9d074c968e57b834" is_pre_election: true
I20260925 15:00:31.542488 21303 raft_consensus.cc:2454] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff6fca8aa31f42feb634a94b9bee7ac4 in term 0.
I20260925 15:00:31.542619 21302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "ff6fca8aa31f42feb634a94b9bee7ac4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82ba8f914e34dcd9d074c968e57b834"
I20260925 15:00:31.542676 21302 raft_consensus.cc:3037] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:31.543561 21302 raft_consensus.cc:2454] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff6fca8aa31f42feb634a94b9bee7ac4 in term 1.
I20260925 15:00:31.552933 21528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.529113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41217 (local address 127.8.13.193:41723)
0925 15:00:31.539516 (+ 10403us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.539552 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:31.539624 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:31.539792 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:31.539793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:31.539794 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:31.539822 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.539823 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.541210 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.541211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.551727 (+ 10516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:31.551734 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.552660 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.552661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:31.552733 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.552737 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:31.552789 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:00:31.552844 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10390,"thread_start_us":66,"threads_started":1}
I20260925 15:00:31.552974 21530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.529458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:41723 (local address 127.8.13.194:41217)
0925 15:00:31.535708 (+ 6250us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.535711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.539598 (+ 3887us) server_negotiation.cc:407] Connection header received
0925 15:00:31.539689 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.539690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.539700 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.539740 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:31.539885 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.541114 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.551853 (+ 10739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.552489 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.552574 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.552587 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.552813 (+ 226us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.552872 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6220,"thread_start_us":146,"threads_started":1}
I20260925 15:00:31.553457 21302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac1a39935a3c4c028d6b7872d5b25d1c" candidate_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a82ba8f914e34dcd9d074c968e57b834" is_pre_election: true
I20260925 15:00:31.553601 21302 raft_consensus.cc:2379] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a58aba2e7b74461a9ef98ee9e642d0c in current term 1: Already voted for candidate ff6fca8aa31f42feb634a94b9bee7ac4 in this term.
I20260925 15:00:31.553997 21102 leader_election.cc:302] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a58aba2e7b74461a9ef98ee9e642d0c; no voters: a82ba8f914e34dcd9d074c968e57b834, ff6fca8aa31f42feb634a94b9bee7ac4
I20260925 15:00:31.554183 21522 raft_consensus.cc:2726] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260925 15:00:31.563436 21511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.560016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58596 (local address 127.8.13.195:36345)
0925 15:00:31.560063 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:31.560065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:31.560074 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:31.560117 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:31.560118 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:31.560137 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:31.560206 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:31.560378 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.561131 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.562267 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:31.562521 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:31.562579 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:31.562823 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:31.562973 (+ 150us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:31.563057 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:31.563319 (+ 262us) server_negotiation.cc:299] Negotiation successful
0925 15:00:31.563347 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:00:31.565653 21302 raft_consensus.cc:1261] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 1 FOLLOWER]: Refusing update from remote peer ff6fca8aa31f42feb634a94b9bee7ac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:31.565989 21165 raft_consensus.cc:1261] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 1 FOLLOWER]: Refusing update from remote peer ff6fca8aa31f42feb634a94b9bee7ac4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:31.575508 21524 consensus_queue.cc:1035] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a82ba8f914e34dcd9d074c968e57b834" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 41217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:31.575767 21531 consensus_queue.cc:1035] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a58aba2e7b74461a9ef98ee9e642d0c" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 33491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260925 15:00:31.580504 21350 tablet.cc:2369] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:00:31.590905 21537 mvcc.cc:205] Tried to move back new op lower bound from 7333267175689220096 to 7333267175575511040. Current Snapshot: MvccSnapshot[applied={T|T < 7333267175689220096}]
I20260925 15:00:31.594162 21017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:31.594285 21017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:31.595144 21017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d5a13faf34243abab647743a7a5fec6: Sending DeleteTablet for 3 replicas of tablet ac1a39935a3c4c028d6b7872d5b25d1c
I20260925 15:00:31.595691 21417 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:31 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:58588
I20260925 15:00:31.595675 21138 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:31 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:36998
I20260925 15:00:31.595997 21283 tablet_service.cc:1514] Processing DeleteTablet for tablet ac1a39935a3c4c028d6b7872d5b25d1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:31 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:35796
I20260925 15:00:31.596138 21546 tablet_replica.cc:330] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c: stopping tablet replica
I20260925 15:00:31.596297 21546 raft_consensus.cc:2229] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:31.596472 21546 raft_consensus.cc:2258] T ac1a39935a3c4c028d6b7872d5b25d1c P 2a58aba2e7b74461a9ef98ee9e642d0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:31.596632 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21056
I20260925 15:00:31.598174 21545 tablet_replica.cc:330] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: stopping tablet replica
I20260925 15:00:31.598388 21545 raft_consensus.cc:2229] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:00:31.598611 21545 raft_consensus.cc:2258] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:31.599171 21545 ts_tablet_manager.cc:1905] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:31.601263 21545 ts_tablet_manager.cc:1918] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:31.601424 21545 log.cc:1196] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/ac1a39935a3c4c028d6b7872d5b25d1c
I20260925 15:00:31.601807 21545 ts_tablet_manager.cc:1939] T ac1a39935a3c4c028d6b7872d5b25d1c P ff6fca8aa31f42feb634a94b9bee7ac4: Deleting consensus metadata
I20260925 15:00:31.602432 21001 catalog_manager.cc:4841] TS ff6fca8aa31f42feb634a94b9bee7ac4 (127.8.13.195:36345): tablet ac1a39935a3c4c028d6b7872d5b25d1c (table test-table [id=f82fe3dae5744387b27b528f39228a9e]) successfully deleted
I20260925 15:00:31.603775 21547 tablet_replica.cc:330] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834: stopping tablet replica
W20260925 15:00:31.603801 21001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:33491 (error 108)
I20260925 15:00:31.604019 21547 raft_consensus.cc:2229] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:31.604246 21547 raft_consensus.cc:2258] T ac1a39935a3c4c028d6b7872d5b25d1c P a82ba8f914e34dcd9d074c968e57b834 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:31.604488 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21219
I20260925 15:00:31.610718 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21352
I20260925 15:00:31.613016 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.612854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:33491 (local address 127.0.0.1:37002)
0925 15:00:31.612918 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.612966 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:33491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260925 15:00:31.613173 21001 catalog_manager.cc:4584] TS 2a58aba2e7b74461a9ef98ee9e642d0c (127.8.13.193:33491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ac1a39935a3c4c028d6b7872d5b25d1c: Network error: Client connection negotiation failed: client connection to 127.8.13.193:33491: connect: Connection refused (error 111)
I20260925 15:00:31.613556 21512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:31.613435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:41217 (local address 127.0.0.1:35804)
0925 15:00:31.613485 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:31.613510 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:41217: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:00:31.616153 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 20984
I20260925 15:00:31.621599 8247 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260925 15:00:31.956378 8247 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260925 15:00:31.956470 8247 mini_ranger.cc:66] Stopping Ranger...
I20260925 15:00:32.291646 8247 mini_ranger.cc:68] Stopped Ranger
2026-09-25 15:00:32.291 UTC [20050] LOG: received smart shutdown request
2026-09-25 15:00:32.295 UTC [20050] LOG: background worker "logical replication launcher" (PID 20059) exited with exit code 1
2026-09-25 15:00:32.295 UTC [20054] LOG: shutting down
2026-09-25 15:00:32.493 UTC [20050] LOG: database system is shut down
2026-09-25T15:00:32Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47579 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:00:32 dist-test-slave-15hz krb5kdc[21557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:00:32 dist-test-slave-15hz krb5kdc[21557](info): set up 2 sockets
Sep 25 15:00:32 dist-test-slave-15hz krb5kdc[21557](info): commencing operation
krb5kdc: starting...
W20260925 15:00:33.535271 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:00:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:00:33.655059 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46315
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:33.735823 21570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:33.736047 21570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:33.736070 21570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:33.737385 21570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:33.737437 21570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:33.737450 21570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:33.737461 21570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:33.737471 21570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:33.739048 21570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:33.739301 21570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:33.739571 21570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:33.744544 21576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:33.744530 21578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:33.744549 21575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:33.744810 21570 server_base.cc:1034] running on GCE node
I20260925 15:00:33.745075 21570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:33.745388 21570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:33.746542 21570 hybrid_clock.cc:648] HybridClock initialized: now 1790348433746545 us; error 58 us; skew 500 ppm
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:33.749292 21570 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:33.749789 21570 webserver.cc:460] Webserver started at http://127.8.13.254:44665/ using document root <none> and password file <none>
I20260925 15:00:33.750012 21570 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:33.750065 21570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:33.750195 21570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:33.751267 21570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "4876de1c27d746218b115749b422dc95"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "7298b7c320a9f1cf328aa207303e84f2"
server_key_iv: "70028ce3189d69e7aba0c23279d489c3"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.751652 21570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "4876de1c27d746218b115749b422dc95"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "7298b7c320a9f1cf328aa207303e84f2"
server_key_iv: "70028ce3189d69e7aba0c23279d489c3"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.752945 21570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:00:33.753762 21585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:33.753969 21570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:00:33.754066 21570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "4876de1c27d746218b115749b422dc95"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "7298b7c320a9f1cf328aa207303e84f2"
server_key_iv: "70028ce3189d69e7aba0c23279d489c3"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.754140 21570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:33.777784 21570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:33.780115 21570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:33.780254 21570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:33.784073 21570 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:46315
I20260925 15:00:33.784116 21637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:46315 every 8 connection(s)
I20260925 15:00:33.784468 21570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:33.785324 21638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:33.786914 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:33.790242 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21570
I20260925 15:00:33.790347 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:00:33.790540 8247 external_mini_cluster.cc:1428] Setting key 58b29de90a83dbe518a0882d1a14aed8
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:33.797551 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.787743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46315 (local address 127.0.0.1:58744)
0925 15:00:33.787916 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.787954 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:33.788482 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:33.788803 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:33.788805 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:33.789120 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:33.789277 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.789281 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.790412 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.790413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.791187 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.791190 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.791779 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.791780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.791855 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.792367 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:33.792375 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:33.793940 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:33.796041 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.796049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.796051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.796255 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.796497 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.796498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.796499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.796591 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.796953 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:33.796959 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:33.797024 (+ 65us) client_negotiation.cc:769] Sending connection context
0925 15:00:33.797111 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:00:33.797356 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":86,"thread_start_us":83,"threads_started":1}
I20260925 15:00:33.797552 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.787924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58744 (local address 127.8.13.254:46315)
0925 15:00:33.788058 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:33.788062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:33.788098 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:00:33.788683 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:33.788685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:33.788707 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:33.788794 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:33.789378 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.790306 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.791268 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.791688 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.791805 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.794035 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:33.794042 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:33.794042 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:33.794049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:33.795938 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.796339 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.796342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.796343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.796430 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.796651 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.796652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.796653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.796887 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:33.796968 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:33.797151 (+ 183us) server_negotiation.cc:299] Negotiation successful
0925 15:00:33.797276 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":42,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.253@KRBTEST.COM; defaulting to no policy
I20260925 15:00:33.798422 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:33.799009 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.798807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39892)
0925 15:00:33.798925 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.798979 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260925 15:00:33.799198 21588 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
I20260925 15:00:33.799702 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.799585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39908)
0925 15:00:33.799630 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.799660 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:00:33.799851 21638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:40179: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
I20260925 15:00:33.816784 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:40179
--webserver_interface=127.8.13.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:33.849119 21638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } attempt: 1
I20260925 15:00:33.849665 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.849527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39916)
0925 15:00:33.849594 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.849637 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:00:33.849992 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.849923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39922)
0925 15:00:33.849956 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.849978 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 15:00:33.850088 21638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:40179: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
W20260925 15:00:33.901146 21645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:33.901371 21645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:33.901401 21645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:33.902586 21645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:33.902643 21645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:33.902661 21645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:33.902678 21645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:33.902693 21645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:33.904346 21645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:40179
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:33.904627 21645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:33.904841 21645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:33.909868 21650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:33.909907 21651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:33.909888 21653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:33.910207 21645 server_base.cc:1034] running on GCE node
I20260925 15:00:33.910409 21645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:33.910679 21645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:33.911819 21645 hybrid_clock.cc:648] HybridClock initialized: now 1790348433911811 us; error 27 us; skew 500 ppm
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:33.914213 21645 init.cc:377] Logged in from keytab as kudu/127.8.13.253@KRBTEST.COM (short username kudu)
I20260925 15:00:33.914613 21645 webserver.cc:460] Webserver started at http://127.8.13.253:38745/ using document root <none> and password file <none>
I20260925 15:00:33.914798 21645 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:33.914837 21645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:33.914925 21645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:33.915997 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data/instance:
uuid: "2050a442b80647fbb7f9af0c2df81482"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "b30de52b3310ff9c07b29f6ed17b197d"
server_key_iv: "53c41e25522d80d8b7acc872d49ceba4"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.916313 21645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal/instance:
uuid: "2050a442b80647fbb7f9af0c2df81482"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "b30de52b3310ff9c07b29f6ed17b197d"
server_key_iv: "53c41e25522d80d8b7acc872d49ceba4"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.917572 21645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:00:33.918257 21638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } attempt: 2
I20260925 15:00:33.918421 21660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:33.918672 21645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:33.918790 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.918645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39924)
0925 15:00:33.918704 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.918752 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:00:33.918797 21645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "2050a442b80647fbb7f9af0c2df81482"
format_stamp: "Formatted at 2026-09-25 15:00:33 on dist-test-slave-15hz"
server_key: "b30de52b3310ff9c07b29f6ed17b197d"
server_key_iv: "53c41e25522d80d8b7acc872d49ceba4"
server_key_version: "encryptionkey@0"
I20260925 15:00:33.918872 21645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:33.919112 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.919027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39940)
0925 15:00:33.919056 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.919080 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 15:00:33.919199 21638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:40179: Network error: Client connection negotiation failed: client connection to 127.8.13.253:40179: connect: Connection refused (error 111)
I20260925 15:00:33.938109 21645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:33.940359 21645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:33.940501 21645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:33.944309 21645 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.253:40179
I20260925 15:00:33.944372 21712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.253:40179 every 8 connection(s)
I20260925 15:00:33.944689 21645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
I20260925 15:00:33.945518 21713 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:33.947156 21713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:33.951758 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21645
I20260925 15:00:33.951843 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-1/wal/instance
I20260925 15:00:33.952025 8247 external_mini_cluster.cc:1428] Setting key 9927cf01193ad5b62d98b544fb513357
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:33.956776 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.947941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58756 (local address 127.8.13.254:46315)
0925 15:00:33.948006 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:33.948008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:33.948228 (+ 220us) server_negotiation.cc:407] Connection header received
0925 15:00:33.948744 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:33.948746 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:33.948759 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:33.948817 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:33.949443 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.949976 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.951009 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.951237 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.951299 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.953890 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:33.953892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:33.953892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:33.953900 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:33.955302 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.955793 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.955795 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.955795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.955852 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.956089 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.956090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.956090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.956207 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:33.956265 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:33.956563 (+ 298us) server_negotiation.cc:299] Negotiation successful
0925 15:00:33.956657 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:00:33.956923 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.947920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46315 (local address 127.0.0.1:58756)
0925 15:00:33.948102 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.948147 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:33.948615 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:33.948889 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:33.948893 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:33.949162 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:33.949370 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.949374 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.950081 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.950082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.950911 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.950914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.951316 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.951317 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.951436 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.952045 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:33.952069 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:33.953772 (+ 1703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:33.955404 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.955414 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.955417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.955718 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.955909 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.955910 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.955910 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.956031 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.956281 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:33.956289 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:33.956434 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:00:33.956525 (+ 91us) client_negotiation.cc:240] Negotiation successful
0925 15:00:33.956691 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":96,"threads_started":1}
I20260925 15:00:33.957502 21713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.13.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.252@KRBTEST.COM" created.
Sep 25 15:00:33 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:00:33.965651 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.957752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39942)
0925 15:00:33.957808 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.957819 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:33.957858 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:33.958152 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:33.958154 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:33.958294 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:33.958359 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.958361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.959346 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.959347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.959902 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:33.959904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.960422 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.960423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:33.960481 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.960878 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:33.960882 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:33.962543 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:33.964656 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.964657 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.964658 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.964811 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.964996 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:33.964996 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:33.964997 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:33.965051 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:33.965313 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:33.965314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:33.965355 (+ 41us) client_negotiation.cc:769] Sending connection context
0925 15:00:33.965415 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:00:33.965528 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:00:33.965721 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.957836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39942 (local address 127.8.13.253:40179)
0925 15:00:33.957989 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:33.957994 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:33.958030 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:00:33.958046 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:33.958048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:33.958059 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:33.958112 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:33.958469 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.959252 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.959987 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:33.960359 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:33.960424 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:33.962661 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:33.962673 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:33.962673 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:33.962680 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:33.964574 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.964881 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.964884 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.964884 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.964933 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:33.965103 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:33.965104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:33.965104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:33.965255 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:33.965325 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:33.965457 (+ 132us) server_negotiation.cc:299] Negotiation successful
0925 15:00:33.965606 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":127,"thread_start_us":101,"threads_started":1}
I20260925 15:00:33.966822 21713 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:33.967204 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.967082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35098)
0925 15:00:33.967126 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.967177 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 15:00:33.967334 21662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
I20260925 15:00:33.967765 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:33.967676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35114)
0925 15:00:33.967714 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:33.967738 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:00:33.967859 21713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:45975: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
I20260925 15:00:33.977818 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:45975
--webserver_interface=127.8.13.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:34.010370 21638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } attempt: 3
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:00:34.017102 21713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } attempt: 1
I20260925 15:00:34.017520 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.017405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35118)
0925 15:00:34.017457 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.017485 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:00:34.017875 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.017753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35124)
0925 15:00:34.017809 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.017845 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260925 15:00:34.018024 21713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:45975: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
I20260925 15:00:34.018288 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.010810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39946)
0925 15:00:34.010883 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.010896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.010960 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.011120 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.011121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.011288 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.011354 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.011356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.012344 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.012345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.012879 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.012881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.013461 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.013462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.013551 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.014113 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.014118 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.015517 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.017143 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.017144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.017145 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.017373 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.017645 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.017646 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.017647 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.017722 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.017998 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.017999 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.018052 (+ 53us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.018101 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.018222 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:00:34.018572 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.010885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.8.13.253:40179)
0925 15:00:34.010956 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.010958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.010971 (+ 13us) server_negotiation.cc:407] Connection header received
0925 15:00:34.011003 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.011004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.011019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.011104 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.011436 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.012233 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.012983 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.013365 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.013451 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.015665 (+ 2214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.015667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.015668 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.015676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.017045 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.017451 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.017453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.017453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.017517 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.017799 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.017800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.017800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.017921 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.018395 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.018441 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.018520 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:00:34.018899 21638 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:34.019254 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.019146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35132)
0925 15:00:34.019201 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.019231 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:00:34.019515 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.019447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35142)
0925 15:00:34.019482 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.019502 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 15:00:34.019610 21638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:45975: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
I20260925 15:00:34.050776 21638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } attempt: 1
I20260925 15:00:34.051250 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.051130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35152)
0925 15:00:34.051185 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.051218 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:00:34.051646 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.051558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35164)
0925 15:00:34.051594 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.051623 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260925 15:00:34.051746 21638 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:45975: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
W20260925 15:00:34.061041 21720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:34.061280 21720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:34.061314 21720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:34.062606 21720 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:34.062660 21720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:34.062680 21720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:34.062695 21720 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:34.062711 21720 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:34.064447 21720 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46315,127.8.13.253:40179,127.8.13.252:45975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:45975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:34.064719 21720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:34.064949 21720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:00:34.069787 21720 server_base.cc:1034] running on GCE node
W20260925 15:00:34.069839 21725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:34.069684 21726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:34.069684 21728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:34.070981 21720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:34.071271 21720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:34.072443 21720 hybrid_clock.cc:648] HybridClock initialized: now 1790348434072423 us; error 27 us; skew 500 ppm
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:34.074982 21720 init.cc:377] Logged in from keytab as kudu/127.8.13.252@KRBTEST.COM (short username kudu)
I20260925 15:00:34.075424 21720 webserver.cc:460] Webserver started at http://127.8.13.252:45219/ using document root <none> and password file <none>
I20260925 15:00:34.075676 21720 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:34.075724 21720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:34.075824 21720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:34.076897 21720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data/instance:
uuid: "a1f3821d891844e6a22855a8e0081bad"
format_stamp: "Formatted at 2026-09-25 15:00:34 on dist-test-slave-15hz"
server_key: "481272d0716c810ed07aa3e53e36fec4"
server_key_iv: "ee78d1cadb3418a6f79b276dddc43cf2"
server_key_version: "encryptionkey@0"
I20260925 15:00:34.077205 21720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal/instance:
uuid: "a1f3821d891844e6a22855a8e0081bad"
format_stamp: "Formatted at 2026-09-25 15:00:34 on dist-test-slave-15hz"
server_key: "481272d0716c810ed07aa3e53e36fec4"
server_key_iv: "ee78d1cadb3418a6f79b276dddc43cf2"
server_key_version: "encryptionkey@0"
I20260925 15:00:34.078547 21720 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:00:34.079336 21735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:34.079562 21720 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:34.079662 21720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "a1f3821d891844e6a22855a8e0081bad"
format_stamp: "Formatted at 2026-09-25 15:00:34 on dist-test-slave-15hz"
server_key: "481272d0716c810ed07aa3e53e36fec4"
server_key_iv: "ee78d1cadb3418a6f79b276dddc43cf2"
server_key_version: "encryptionkey@0"
I20260925 15:00:34.079727 21720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:34.086194 21713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } attempt: 2
I20260925 15:00:34.086692 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.086532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35176)
0925 15:00:34.086611 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.086657 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:00:34.087054 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.086963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35182)
0925 15:00:34.087008 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.087031 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:00:34.087138 21713 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:45975: Network error: Client connection negotiation failed: client connection to 127.8.13.252:45975: connect: Connection refused (error 111)
I20260925 15:00:34.092948 21720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:34.095134 21720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:34.095247 21720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:34.098999 21720 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.252:45975
I20260925 15:00:34.099031 21787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.252:45975 every 8 connection(s)
I20260925 15:00:34.099385 21720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
I20260925 15:00:34.100270 21788 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:34.101927 21788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:34.102490 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21720
I20260925 15:00:34.102578 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790348284423390-8247-0/minicluster-data/master-2/wal/instance
I20260925 15:00:34.102730 8247 external_mini_cluster.cc:1428] Setting key 623858fa5b46ab24fa5089cf141cd4ee
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:00:34.111007 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.102731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58764 (local address 127.8.13.254:46315)
0925 15:00:34.102800 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.102803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.103039 (+ 236us) server_negotiation.cc:407] Connection header received
0925 15:00:34.103623 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.103625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.103637 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.103680 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.104224 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.104733 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.105625 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.105832 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.105890 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.108096 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.108097 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.108097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.108104 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.109542 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.110027 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.110028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.110028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.110085 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.110315 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.110316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.110317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.110456 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.110515 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.110828 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.110913 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:00:34.111233 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.103055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35192 (local address 127.8.13.252:45975)
0925 15:00:34.103182 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.103187 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.103317 (+ 130us) server_negotiation.cc:407] Connection header received
0925 15:00:34.103506 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.103512 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.103556 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.103640 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.104274 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.105079 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.105804 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.106108 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.106165 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.108534 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.108541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.108542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.108549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.110094 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.110417 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.110420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.110420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.110465 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.110638 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.110639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.110639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.110787 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.110920 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.110990 (+ 70us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.111054 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":62,"threads_started":1}
I20260925 15:00:34.111267 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.102720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46315 (local address 127.0.0.1:58764)
0925 15:00:34.102905 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.102955 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.103506 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.103708 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.103711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.103973 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.104132 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.104136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.104806 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.104807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.105552 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.105555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.105904 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.105905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.106056 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.106574 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.106582 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.108011 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.109626 (+ 1615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.109631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.109633 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.109953 (+ 320us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.110142 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.110143 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.110143 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.110259 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.110516 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.110521 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.110670 (+ 149us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.110791 (+ 121us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.110947 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260925 15:00:34.112104 21788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:34.118554 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.112343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:40179 (local address 127.0.0.1:39954)
0925 15:00:34.112383 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.112394 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.112427 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.112568 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.112569 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.112671 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.112727 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.112729 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.113547 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.113547 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.114036 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.114038 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.114341 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.114342 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.114418 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.114856 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.114861 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.116208 (+ 1347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.117585 (+ 1377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.117586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.117587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.117744 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.117949 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.117950 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.117950 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.118005 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.118247 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.118248 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.118294 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.118348 (+ 54us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.118448 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 15:00:34.118585 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.112424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.8.13.253:40179)
0925 15:00:34.112470 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.112472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.112479 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:34.112503 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.112504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.112513 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.112570 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.112781 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.113484 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.114092 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.114269 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.114318 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.116317 (+ 1999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.116319 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.116319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.116326 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.117514 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.117824 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.117825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.117826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.117885 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.118065 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.118065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.118065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.118184 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.118237 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.118381 (+ 144us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.118451 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:00:34.119019 21788 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:00:34.120314 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.112091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.8.13.254:46315)
0925 15:00:34.112124 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.112126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.112133 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:34.112175 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.112176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.112183 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.112230 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.112753 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.113484 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.114821 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.115134 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.115191 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.117894 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.117895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.117895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.117901 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.119306 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.119634 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.119635 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.119635 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.119687 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.119842 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.119843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.119843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.119946 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.120120 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.120156 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.120225 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348434, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:00:34.125270 21790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.119270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35204)
0925 15:00:34.119309 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.119319 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.119355 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.119459 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.119460 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.119616 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.119649 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.119651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.120228 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.120229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.120931 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.120934 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.121309 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.121310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.121355 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.121719 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.121723 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.122815 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.124367 (+ 1552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.124367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.124368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.124515 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.124683 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.124683 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.124683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.124763 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.124973 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.124974 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.125010 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.125070 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.125165 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:00:34.125319 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.119319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35204 (local address 127.8.13.252:45975)
0925 15:00:34.119352 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.119353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.119360 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:34.119405 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.119406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.119417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.119458 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.119710 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.120167 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.120997 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.121254 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.121306 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.122883 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.122884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.122885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.122891 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.124295 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.124583 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.124584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.124584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.124633 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.124814 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.124815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.124815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.124927 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.124982 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.125096 (+ 114us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.125219 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":19}
I20260925 15:00:34.126860 21788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: Bootstrap starting.
I20260925 15:00:34.126906 21638 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } attempt: 2
I20260925 15:00:34.127924 21788 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:34.128225 21788 log.cc:824] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:34.129055 21788 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: No bootstrap required, opened a new log
I20260925 15:00:34.129451 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.120918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.8.13.253:40179)
0925 15:00:34.120968 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.120970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.121004 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:00:34.121068 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.121069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.121079 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.121121 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.121539 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.122016 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.122655 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.122938 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.122991 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.125217 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.125218 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.125218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.125226 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.128150 (+ 2924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.128534 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.128536 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.128536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.128603 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.128833 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.128834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.128835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.128950 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.129251 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.129296 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.129376 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:00:34.130959 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:00:34.131122 21788 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.131286 21788 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:34.131319 21788 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1f3821d891844e6a22855a8e0081bad, State: Initialized, Role: FOLLOWER
I20260925 15:00:34.131444 21788 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.131763 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.131816 21788 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:34.131839 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.134234 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.127227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.8.13.252:45975)
0925 15:00:34.127262 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.127263 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.127272 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:34.127342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.127342 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.127350 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.127387 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.127720 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.128431 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.129357 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.129624 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.129907 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.131436 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.131437 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.131437 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.131444 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.133016 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.133422 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.133423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.133423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.133481 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.133693 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.133693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.133694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.133854 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.133910 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.134082 (+ 172us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.134149 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:00:34.134238 21640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.127175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35218)
0925 15:00:34.127233 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.127243 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.127282 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.127396 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.127397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.127590 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.127638 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.127640 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.128528 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.128529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.129268 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.129272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.129705 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.129706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.129760 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.130093 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.130098 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.131352 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.133095 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.133096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.133096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.133317 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.133568 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.133569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.133569 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.133631 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.133936 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.133937 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.133982 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.134043 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.134138 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:00:34.135763 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: Bootstrap starting.
I20260925 15:00:34.133751 21720 master_runner.cc:186] Error getting master registration for 127.8.13.254:46315: OK, instance_id { permanent_uuid: "4876de1c27d746218b115749b422dc95" instance_seqno: 1790348433782850 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:00:34.136475 21638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:34.136524 21720 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:00:34.136768 21638 log.cc:824] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:34.137548 21638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: No bootstrap required, opened a new log
I20260925 15:00:34.139454 21638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.139662 21638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:34.139703 21638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4876de1c27d746218b115749b422dc95, State: Initialized, Role: FOLLOWER
W20260925 15:00:34.139842 21810 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260925 15:00:34.139824 21638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
W20260925 15:00:34.139892 21810 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:00:34.140128 21638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:34.141245 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.133444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35220 (local address 127.8.13.252:45975)
0925 15:00:34.134292 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.134294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.134300 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:00:34.134317 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.134317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.134326 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.134366 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.135057 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.135700 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.136608 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.136861 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.136914 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.138261 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.138262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.138262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.138269 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.139863 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.140419 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.140420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.140420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.140476 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.140651 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.140652 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.140652 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.140770 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.140850 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.141075 (+ 225us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.141146 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":836,"thread_start_us":59,"threads_started":1}
I20260925 15:00:34.143613 21641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.133408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58778 (local address 127.8.13.254:46315)
0925 15:00:34.133540 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.133542 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.133549 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:34.133568 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.133568 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.133578 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.133628 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.134283 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.134963 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.136003 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.136265 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.136326 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.138018 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.138020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.138020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.138027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.142314 (+ 4287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.142669 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.142670 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.142670 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.142725 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.142906 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.142906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.142907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.143022 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.143112 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.143421 (+ 309us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.143494 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260925 15:00:34.144726 21827 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:00:34.144766 21827 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:00:34.144805 21717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.133380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39976 (local address 127.8.13.253:40179)
0925 15:00:34.133429 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.137014 (+ 3585us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.137025 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:00:34.137049 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.137050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.137061 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.137117 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.137837 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.138501 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.139421 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.139757 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.139817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.141165 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.141166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.141167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.141174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.143199 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.143642 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.143643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.143644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.143701 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.143881 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.143882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.143882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.144088 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.144187 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.144638 (+ 451us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.144718 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:00:34.144922 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.144994 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.143246 21570 master_runner.cc:186] Error getting master registration for 127.8.13.253:40179: OK, instance_id { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" instance_seqno: 1790348433943207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:00:34.146183 21570 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:00:34.178296 21713 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } attempt: 3
Sep 25 15:00:34 dist-test-slave-15hz krb5kdc[21557](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:00:34.188275 21715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.178604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:45975 (local address 127.0.0.1:35222)
0925 15:00:34.178656 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:34.178671 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:34.178732 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:34.178988 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:34.178989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:34.179200 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:34.179264 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.179267 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.180214 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.180217 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.181021 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:34.181024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.181514 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.181515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:34.181571 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.182161 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:34.182167 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:34.183824 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:34.187084 (+ 3260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.187086 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.187086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.187335 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.187584 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:34.187585 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:34.187586 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:34.187649 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:34.187941 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:34.187942 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:34.187991 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:00:34.188037 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:00:34.188146 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260925 15:00:34.188683 21791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:34.178832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35222 (local address 127.8.13.252:45975)
0925 15:00:34.178871 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:34.178873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:34.178880 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:34.178902 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:34.178903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:34.178912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:34.179344 (+ 432us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:34.179378 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.180102 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.181131 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:34.181432 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:34.181778 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:34.184200 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:34.184202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:34.184202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:34.184212 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:34.186972 (+ 2760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.187419 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.187420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.187421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.187480 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:34.187719 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:34.187719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:34.187720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:34.187866 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:34.188468 (+ 602us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:34.188511 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:00:34.188594 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:00:34.190315 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: Bootstrap starting.
I20260925 15:00:34.191102 21713 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:34.191428 21713 log.cc:824] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:34.192355 21713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: No bootstrap required, opened a new log
I20260925 15:00:34.194506 21713 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.194747 21713 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:34.194834 21713 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2050a442b80647fbb7f9af0c2df81482, State: Initialized, Role: FOLLOWER
I20260925 15:00:34.195019 21713 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.195420 21713 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:34.196480 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.196583 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:00:34.198344 21841 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:00:34.198400 21841 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:00:34.203821 21794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:34.203959 21794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.204280 21794 leader_election.cc:288] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4876de1c27d746218b115749b422dc95 (127.8.13.254:46315), 2050a442b80647fbb7f9af0c2df81482 (127.8.13.253:40179)
I20260925 15:00:34.204916 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1f3821d891844e6a22855a8e0081bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2050a442b80647fbb7f9af0c2df81482" is_pre_election: true
I20260925 15:00:34.205090 21687 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f3821d891844e6a22855a8e0081bad in term 0.
I20260925 15:00:34.205556 21738 leader_election.cc:302] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2050a442b80647fbb7f9af0c2df81482, a1f3821d891844e6a22855a8e0081bad; no voters:
I20260925 15:00:34.205507 21612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1f3821d891844e6a22855a8e0081bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4876de1c27d746218b115749b422dc95" is_pre_election: true
I20260925 15:00:34.205694 21612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1f3821d891844e6a22855a8e0081bad in term 0.
I20260925 15:00:34.206969 21645 master_runner.cc:263] No leader master found from master 2050a442b80647fbb7f9af0c2df81482
I20260925 15:00:34.207033 21645 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:00:34.207137 21794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:34.207202 21794 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:34.207226 21794 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:34.208215 21794 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.208367 21794 leader_election.cc:288] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [CANDIDATE]: Term 1 election: Requested vote from peers 4876de1c27d746218b115749b422dc95 (127.8.13.254:46315), 2050a442b80647fbb7f9af0c2df81482 (127.8.13.253:40179)
I20260925 15:00:34.208537 21612 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1f3821d891844e6a22855a8e0081bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4876de1c27d746218b115749b422dc95"
I20260925 15:00:34.208577 21687 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a1f3821d891844e6a22855a8e0081bad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2050a442b80647fbb7f9af0c2df81482"
I20260925 15:00:34.208653 21687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:34.208647 21612 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:34.209585 21687 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f3821d891844e6a22855a8e0081bad in term 1.
I20260925 15:00:34.209650 21612 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1f3821d891844e6a22855a8e0081bad in term 1.
I20260925 15:00:34.209777 21738 leader_election.cc:302] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2050a442b80647fbb7f9af0c2df81482, a1f3821d891844e6a22855a8e0081bad; no voters:
I20260925 15:00:34.210003 21794 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:34.210177 21794 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [term 1 LEADER]: Becoming Leader. State: Replica: a1f3821d891844e6a22855a8e0081bad, State: Running, Role: LEADER
I20260925 15:00:34.210278 21794 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } }
I20260925 15:00:34.210698 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f3821d891844e6a22855a8e0081bad. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.210775 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:34.216192 21845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:34.216414 21845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:34.225090 21612 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [term 1 FOLLOWER]: Refusing update from remote peer a1f3821d891844e6a22855a8e0081bad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:34.225090 21687 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [term 1 FOLLOWER]: Refusing update from remote peer a1f3821d891844e6a22855a8e0081bad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:34.225449 21843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [LEADER]: Connected to new peer: Peer: permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:34.225648 21794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [LEADER]: Connected to new peer: Peer: permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:34.226818 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f3821d891844e6a22855a8e0081bad. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.226817 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1f3821d891844e6a22855a8e0081bad. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.226925 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.226925 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.232915 21843 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.233058 21843 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:34.233049 21816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.233167 21816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.233320 21846 mvcc.cc:205] Tried to move back new op lower bound from 7333267186583523328 to 7333267186526023680. Current Snapshot: MvccSnapshot[applied={T|T < 7333267186583523328 or (T in {7333267186583523328})}]
I20260925 15:00:34.233510 21845 catalog_manager.cc:1347] Generated new cluster ID: 1aae746d12f549b7b46649c1575d1867
I20260925 15:00:34.233557 21845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:00:34.242574 21794 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.242712 21794 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:34.243453 21830 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a1f3821d891844e6a22855a8e0081bad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4876de1c27d746218b115749b422dc95" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46315 } } peers { permanent_uuid: "2050a442b80647fbb7f9af0c2df81482" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 40179 } } peers { permanent_uuid: "a1f3821d891844e6a22855a8e0081bad" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 45975 } } }
I20260925 15:00:34.243584 21830 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:00:34.244133 21845 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:34.244874 21845 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:34.247639 21848 mvcc.cc:205] Tried to move back new op lower bound from 7333267186583523328 to 7333267186526023680. Current Snapshot: MvccSnapshot[applied={T|T < 7333267186583523328}]
I20260925 15:00:34.248798 21847 mvcc.cc:205] Tried to move back new op lower bound from 7333267186583523328 to 7333267186526023680. Current Snapshot: MvccSnapshot[applied={T|T < 7333267186583523328}]
I20260925 15:00:34.254815 21845 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1f3821d891844e6a22855a8e0081bad: Generated new TSK 0
I20260925 15:00:34.255050 21845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:00:35.145184 21827 catalog_manager.cc:1259] Loaded cluster ID: 1aae746d12f549b7b46649c1575d1867
I20260925 15:00:35.145268 21827 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: loading cluster ID for follower catalog manager: success
I20260925 15:00:35.146037 21827 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: acquiring CA information for follower catalog manager: success
I20260925 15:00:35.146339 21827 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4876de1c27d746218b115749b422dc95: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:00:35.198830 21841 catalog_manager.cc:1259] Loaded cluster ID: 1aae746d12f549b7b46649c1575d1867
I20260925 15:00:35.198912 21841 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: loading cluster ID for follower catalog manager: success
I20260925 15:00:35.199771 21841 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: acquiring CA information for follower catalog manager: success
I20260925 15:00:35.200076 21841 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2050a442b80647fbb7f9af0c2df81482: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:00:35.314005 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21570
I20260925 15:00:35.318780 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21645
I20260925 15:00:35.323087 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21720
2026-09-25T15:00:35Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2737 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:00:35 dist-test-slave-15hz krb5kdc[21864](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:00:35 dist-test-slave-15hz krb5kdc[21864](info): set up 2 sockets
Sep 25 15:00:35 dist-test-slave-15hz krb5kdc[21864](info): commencing operation
krb5kdc: starting...
W20260925 15:00:36.283394 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.942s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:00:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:00:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:00:36.398459 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:38583
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:40709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:38583
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:36.479393 21877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:36.479773 21877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:36.479825 21877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:36.481506 21877 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:36.481565 21877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:36.481587 21877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:36.481603 21877 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:36.481621 21877 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:36.483247 21877 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:40709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:38583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:38583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:36.483494 21877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:36.483770 21877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:36.488512 21885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:36.488499 21883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:36.488499 21882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:36.488760 21877 server_base.cc:1034] running on GCE node
I20260925 15:00:36.489010 21877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:36.489300 21877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:36.490447 21877 hybrid_clock.cc:648] HybridClock initialized: now 1790348436490431 us; error 31 us; skew 500 ppm
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21864](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348436, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:36.492902 21877 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:36.493319 21877 webserver.cc:460] Webserver started at http://127.8.13.254:40929/ using document root <none> and password file <none>
I20260925 15:00:36.493494 21877 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:36.493531 21877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:36.493633 21877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:36.494661 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "de02325c07ab4790a7a49a70fe449b60"
format_stamp: "Formatted at 2026-09-25 15:00:36 on dist-test-slave-15hz"
server_key: "6d9d57d61f5d9047f21c996edca18ecd"
server_key_iv: "1a93e4872bb108e9f3d010cd92915832"
server_key_version: "encryptionkey@0"
I20260925 15:00:36.494954 21877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "de02325c07ab4790a7a49a70fe449b60"
format_stamp: "Formatted at 2026-09-25 15:00:36 on dist-test-slave-15hz"
server_key: "6d9d57d61f5d9047f21c996edca18ecd"
server_key_iv: "1a93e4872bb108e9f3d010cd92915832"
server_key_version: "encryptionkey@0"
I20260925 15:00:36.496214 21877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:00:36.496836 21892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:36.496984 21877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:00:36.497066 21877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "de02325c07ab4790a7a49a70fe449b60"
format_stamp: "Formatted at 2026-09-25 15:00:36 on dist-test-slave-15hz"
server_key: "6d9d57d61f5d9047f21c996edca18ecd"
server_key_iv: "1a93e4872bb108e9f3d010cd92915832"
server_key_version: "encryptionkey@0"
I20260925 15:00:36.497124 21877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:36.517976 21877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:36.519953 21877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:36.520064 21877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:36.523684 21877 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:38583
I20260925 15:00:36.523732 21944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:38583 every 8 connection(s)
I20260925 15:00:36.524048 21877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:36.524880 21945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:36.527478 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60: Bootstrap starting.
I20260925 15:00:36.528054 21945 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:36.528327 21945 log.cc:824] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:36.529024 21945 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60: No bootstrap required, opened a new log
I20260925 15:00:36.530553 21945 raft_consensus.cc:348] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } }
I20260925 15:00:36.530692 21945 raft_consensus.cc:374] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:36.530725 21945 raft_consensus.cc:729] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de02325c07ab4790a7a49a70fe449b60, State: Initialized, Role: FOLLOWER
I20260925 15:00:36.530864 21945 consensus_queue.cc:260] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } }
I20260925 15:00:36.530938 21945 raft_consensus.cc:388] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:36.530978 21945 raft_consensus.cc:482] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:36.531025 21945 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:36.531517 21945 raft_consensus.cc:504] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } }
I20260925 15:00:36.531656 21945 leader_election.cc:302] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de02325c07ab4790a7a49a70fe449b60; no voters:
I20260925 15:00:36.531807 21945 leader_election.cc:288] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:00:36.531849 21948 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:36.531994 21948 raft_consensus.cc:686] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [term 1 LEADER]: Becoming Leader. State: Replica: de02325c07ab4790a7a49a70fe449b60, State: Running, Role: LEADER
I20260925 15:00:36.532047 21945 sys_catalog.cc:564] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:36.532081 21948 consensus_queue.cc:237] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } }
I20260925 15:00:36.532460 21950 sys_catalog.cc:455] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [sys.catalog]: SysCatalogTable state changed. Reason: New leader de02325c07ab4790a7a49a70fe449b60. Latest consensus state: current_term: 1 leader_uuid: "de02325c07ab4790a7a49a70fe449b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } } }
I20260925 15:00:36.532539 21950 sys_catalog.cc:458] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:36.532657 21949 sys_catalog.cc:455] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de02325c07ab4790a7a49a70fe449b60" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de02325c07ab4790a7a49a70fe449b60" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 38583 } } }
I20260925 15:00:36.532792 21949 sys_catalog.cc:458] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:36.532744 21957 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:36.533179 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21877
I20260925 15:00:36.533277 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:00:36.533488 8247 external_mini_cluster.cc:1428] Setting key 47b77dfc3577ba6dd836b344f68ba4e7
I20260925 15:00:36.533484 21957 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:36.535059 21957 catalog_manager.cc:1347] Generated new cluster ID: c9ff252265fc40c39ad9ab4509af2854
I20260925 15:00:36.535109 21957 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21864](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:36.545097 21967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:36.534058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49016 (local address 127.8.13.254:38583)
0925 15:00:36.534475 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:36.534482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:36.534506 (+ 24us) server_negotiation.cc:407] Connection header received
0925 15:00:36.534637 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:36.534645 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:36.534711 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:36.534836 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:36.535563 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:36.536958 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:36.538007 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:36.538627 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:36.538711 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:36.541413 (+ 2702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:36.541417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:36.541417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:36.541429 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:36.543255 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:36.543640 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:36.543643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:36.543643 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:36.543751 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:36.543917 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:36.543917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:36.543918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:36.544191 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:36.544328 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:36.544730 (+ 402us) server_negotiation.cc:299] Negotiation successful
0925 15:00:36.544861 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":61,"threads_started":1}
I20260925 15:00:36.545943 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:00:36.547523 21957 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:36.548065 21957 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:36.552717 21957 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de02325c07ab4790a7a49a70fe449b60: Generated new TSK 0
I20260925 15:00:36.552888 21957 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:00:36.600116 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21877
2026-09-25T15:00:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1275 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21972](info): set up 2 sockets
Sep 25 15:00:36 dist-test-slave-15hz krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260925 15:00:37.547426 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:00:37 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:00:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:00:37.658731 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:37.739920 21985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:37.740177 21985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:37.740214 21985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:37.741442 21985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:37.741631 21985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:37.741710 21985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:37.741734 21985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:37.741755 21985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:37.743590 21985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21985
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:37.743870 21985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:37.744081 21985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:00:37.748898 21985 server_base.cc:1034] running on GCE node
W20260925 15:00:37.748879 21993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:37.748865 21991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:37.748880 21990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:37.749416 21985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:37.749825 21985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:37.751032 21985 hybrid_clock.cc:648] HybridClock initialized: now 1790348437751008 us; error 46 us; skew 500 ppm
Sep 25 15:00:37 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:37.753795 21985 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:37.754294 21985 webserver.cc:460] Webserver started at http://127.8.13.254:37107/ using document root <none> and password file <none>
I20260925 15:00:37.754501 21985 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:37.754549 21985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:37.754678 21985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:37.755861 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "22538318a0654c13b0daac23579521b3"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "d63889d4e57167eb8c366211162dec44"
server_key_iv: "8431b84c39077d7555aa3e1e878edf31"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.756202 21985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "22538318a0654c13b0daac23579521b3"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "d63889d4e57167eb8c366211162dec44"
server_key_iv: "8431b84c39077d7555aa3e1e878edf31"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.757555 21985 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:00:37.758363 22000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:37.758548 21985 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:37.758648 21985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "22538318a0654c13b0daac23579521b3"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "d63889d4e57167eb8c366211162dec44"
server_key_iv: "8431b84c39077d7555aa3e1e878edf31"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.758720 21985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:37.790848 21985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:37.792996 21985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:37.793115 21985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:37.796981 21985 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42091
I20260925 15:00:37.797019 22052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42091 every 8 connection(s)
I20260925 15:00:37.797345 21985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:37.798291 22053 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:37.800649 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap starting.
I20260925 15:00:37.801223 22053 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:37.801468 22053 log.cc:824] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:37.802167 22053 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: No bootstrap required, opened a new log
I20260925 15:00:37.803764 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 21985
I20260925 15:00:37.803850 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:00:37.803838 22053 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:37.804004 22053 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:37.804042 8247 external_mini_cluster.cc:1428] Setting key fc12a3fecf5b4dc1a61c483b3c07c66e
I20260925 15:00:37.804049 22053 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22538318a0654c13b0daac23579521b3, State: Initialized, Role: FOLLOWER
I20260925 15:00:37.804181 22053 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:37.804280 22053 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:37.804312 22053 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:37.804360 22053 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:37.805403 22053 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:37.805498 22053 leader_election.cc:302] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22538318a0654c13b0daac23579521b3; no voters:
I20260925 15:00:37.805650 22053 leader_election.cc:288] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:00:37.805697 22058 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:37.805879 22058 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 LEADER]: Becoming Leader. State: Replica: 22538318a0654c13b0daac23579521b3, State: Running, Role: LEADER
I20260925 15:00:37.805894 22053 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:37.806030 22058 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:37.806372 22060 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22538318a0654c13b0daac23579521b3. Latest consensus state: current_term: 1 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:37.806382 22059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:37.806541 22059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:37.806718 22060 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:37.807080 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:37.807646 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:37.808987 22067 catalog_manager.cc:1347] Generated new cluster ID: fb2b1016edae49a5916322f39693e4f5
I20260925 15:00:37.809036 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:00:37 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:37.815678 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:37.804533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50316 (local address 127.8.13.254:42091)
0925 15:00:37.804703 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:37.804721 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:37.804760 (+ 39us) server_negotiation.cc:407] Connection header received
0925 15:00:37.804892 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:37.804899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:37.804960 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:37.805115 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:37.805786 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:37.807245 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:37.808406 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:37.809000 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:37.809141 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:37.811889 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:37.811893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:37.811894 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:37.811908 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:37.813879 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:37.814321 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:37.814325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:37.814325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:37.814468 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:37.814670 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:37.814671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:37.814671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:37.814980 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:37.815148 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:37.815258 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:00:37.815400 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
I20260925 15:00:37.816344 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:37.816898 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:37.828823 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Generated new TSK 0
I20260925 15:00:37.829128 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:00:37.859589 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:37.955987 22079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:37.956228 22079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:37.956254 22079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:37.957659 22079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:37.957722 22079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:37.957752 22079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:37.959452 22079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:37.959779 22079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:37.960037 22079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:37.965258 22084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:37.965274 22085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:37.965318 22087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:37.965462 22079 server_base.cc:1034] running on GCE node
I20260925 15:00:37.965683 22079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:37.965926 22079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:37.967057 22079 hybrid_clock.cc:648] HybridClock initialized: now 1790348437967047 us; error 29 us; skew 500 ppm
Sep 25 15:00:37 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:37.969543 22079 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:37.970023 22079 webserver.cc:460] Webserver started at http://127.8.13.193:35785/ using document root <none> and password file <none>
I20260925 15:00:37.970247 22079 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:37.970293 22079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:37.970422 22079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:37.971477 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "ab472316eb694e3888f1ceb444891201"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "396edad3861e4fe823421e33bba471d8"
server_key_iv: "f22732c657ffaf2e6f7e00df51ed5deb"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.971871 22079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "ab472316eb694e3888f1ceb444891201"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "396edad3861e4fe823421e33bba471d8"
server_key_iv: "f22732c657ffaf2e6f7e00df51ed5deb"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.973188 22079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:37.973903 22094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:37.974088 22079 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:37.974179 22079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ab472316eb694e3888f1ceb444891201"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "396edad3861e4fe823421e33bba471d8"
server_key_iv: "f22732c657ffaf2e6f7e00df51ed5deb"
server_key_version: "encryptionkey@0"
I20260925 15:00:37.974236 22079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.001519 22079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.003463 22079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.003652 22079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.003870 22079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:38.004165 22079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:38.004195 22079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.004217 22079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:38.004244 22079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.009918 22079 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:37261
I20260925 15:00:38.009981 22207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:37261 every 8 connection(s)
I20260925 15:00:38.010291 22079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:38.015661 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22079
I20260925 15:00:38.015762 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:00:38.015960 8247 external_mini_cluster.cc:1428] Setting key 1344f0f9ac3465c209683419918e5bf2
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:38.019884 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.011026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:47115 (local address 127.8.13.254:42091)
0925 15:00:38.011083 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.011085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.011333 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:00:38.011803 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.011805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.011819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.011875 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.012793 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.013327 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.014309 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.014521 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.014581 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.016966 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.016968 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.016968 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.016977 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.018484 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.018926 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.018927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.018928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.018993 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.019205 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.019205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.019206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.019341 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.019404 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.019686 (+ 282us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.019787 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:00:38.019979 22210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.011009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:47115)
0925 15:00:38.011181 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.011226 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.011721 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.011942 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.011945 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.012507 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.012706 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.012713 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.013397 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.013398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.014227 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.014229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.014588 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.014589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.014691 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.015213 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.015222 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.016872 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.018597 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.018606 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.018609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.018865 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.019054 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.019055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.019055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.019154 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.019394 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.019397 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.019548 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.019658 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.019794 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
I20260925 15:00:38.020457 22208 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:38.020583 22208 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:38.020792 22208 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:38.021334 22017 ts_manager.cc:194] Registered new tserver with Master: ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261)
I20260925 15:00:38.022014 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:47115
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:00:38.040908 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:38.122498 22213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.122728 22213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.122753 22213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.123970 22213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.124013 22213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.124040 22213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:38.125514 22213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.125756 22213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.125981 22213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.130734 22221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.130759 22218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.130854 22213 server_base.cc:1034] running on GCE node
W20260925 15:00:38.130730 22219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.131232 22213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.131461 22213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.132596 22213 hybrid_clock.cc:648] HybridClock initialized: now 1790348438132594 us; error 43 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.134948 22213 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:38.135370 22213 webserver.cc:460] Webserver started at http://127.8.13.194:34175/ using document root <none> and password file <none>
I20260925 15:00:38.135583 22213 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.135629 22213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.135733 22213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:38.136773 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "b96f4a705297e623861e45b01170c0dd"
server_key_iv: "9cb296dc27dc4dc28bfebcac9c52d110"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.137069 22213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "b96f4a705297e623861e45b01170c0dd"
server_key_iv: "9cb296dc27dc4dc28bfebcac9c52d110"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.138335 22213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:00:38.139076 22228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.139269 22213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:38.139355 22213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "b96f4a705297e623861e45b01170c0dd"
server_key_iv: "9cb296dc27dc4dc28bfebcac9c52d110"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.139415 22213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.152235 22213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.154222 22213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.154340 22213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.154529 22213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:38.154841 22213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:38.154881 22213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.154902 22213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:38.154913 22213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.160260 22213 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:37761
I20260925 15:00:38.160315 22341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:37761 every 8 connection(s)
I20260925 15:00:38.160629 22213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:38.165752 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22213
I20260925 15:00:38.165863 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:00:38.166044 8247 external_mini_cluster.cc:1428] Setting key 9345605a78bdcc09ac346f9a3b5aeaf7
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:38.170331 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.161401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:43413 (local address 127.8.13.254:42091)
0925 15:00:38.161469 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.161472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.161757 (+ 285us) server_negotiation.cc:407] Connection header received
0925 15:00:38.162295 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.162296 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.162307 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.162359 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.163365 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.163983 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.164949 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.165135 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.165179 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.167443 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.167444 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.167445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.167453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.168987 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.169375 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.169376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.169376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.169438 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.169653 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.169654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.169655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.169771 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.169825 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.170142 (+ 317us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.170235 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:00:38.170526 22344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.161365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:43413)
0925 15:00:38.161600 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.161657 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.162210 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.162424 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.162428 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.163071 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.163277 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.163283 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.164051 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.164051 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.164884 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.164886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.165190 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.165190 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.165285 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.165810 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.165818 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.167349 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.169065 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.169081 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.169083 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.169308 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.169492 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.169492 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.169493 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.169601 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.169823 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.169830 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.169996 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.170106 (+ 110us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.170300 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260925 15:00:38.171022 22342 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:38.171139 22342 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:38.171346 22342 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:38.171855 22017 ts_manager.cc:194] Registered new tserver with Master: b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
I20260925 15:00:38.172420 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:43413
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:00:38.190557 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:38.271287 22347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.271516 22347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.271584 22347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.272835 22347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.272886 22347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.272924 22347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:38.274523 22347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.274797 22347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.275029 22347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.280309 22353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.280368 22347 server_base.cc:1034] running on GCE node
W20260925 15:00:38.280331 22352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.280320 22355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.280776 22347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.281002 22347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.282173 22347 hybrid_clock.cc:648] HybridClock initialized: now 1790348438282159 us; error 30 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.284842 22347 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:38.285283 22347 webserver.cc:460] Webserver started at http://127.8.13.195:37127/ using document root <none> and password file <none>
I20260925 15:00:38.285485 22347 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.285543 22347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.285669 22347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:38.286736 22347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "658742a535bb49f486a7fc9bc17f2c50"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "944d27d2081e5f9df4806e0f3a9a97d5"
server_key_iv: "19feabef80bcc910060a59fa575a353e"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.287076 22347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "658742a535bb49f486a7fc9bc17f2c50"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "944d27d2081e5f9df4806e0f3a9a97d5"
server_key_iv: "19feabef80bcc910060a59fa575a353e"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.288465 22347 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:00:38.289288 22362 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.289450 22347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:38.289541 22347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "658742a535bb49f486a7fc9bc17f2c50"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "944d27d2081e5f9df4806e0f3a9a97d5"
server_key_iv: "19feabef80bcc910060a59fa575a353e"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.289606 22347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.324611 22347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.326797 22347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.326937 22347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.327165 22347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:38.327458 22347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:38.327491 22347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.327521 22347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:38.327574 22347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.333060 22347 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:42585
I20260925 15:00:38.333112 22475 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:42585 every 8 connection(s)
I20260925 15:00:38.333441 22347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:38.335752 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22347
I20260925 15:00:38.335824 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:00:38.335948 8247 external_mini_cluster.cc:1428] Setting key be670df8223475b7deaa442510b0bdff
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:38.342492 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.334179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:59587 (local address 127.8.13.254:42091)
0925 15:00:38.334251 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.334253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.334440 (+ 187us) server_negotiation.cc:407] Connection header received
0925 15:00:38.334921 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.334923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.334935 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.334998 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.336057 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.336712 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.337697 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.337897 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.337951 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.339989 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.339991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.339991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.339999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.341166 (+ 1167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.341562 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.341563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.341563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.341620 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.341834 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.341835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.341835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.341952 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.342007 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.342315 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.342399 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260925 15:00:38.342679 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.334139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:59587)
0925 15:00:38.334293 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.334335 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.334817 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.335053 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.335056 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.335682 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.335935 (+ 253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.335941 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.336796 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.336797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.337617 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.337619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.337965 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.337966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.338084 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.338540 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.338549 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.339900 (+ 1351us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.341250 (+ 1350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.341258 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.341261 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.341479 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.341673 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.341674 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.341675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.341775 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.342006 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.342012 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.342181 (+ 169us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.342281 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.342446 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260925 15:00:38.343097 22476 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:38.343222 22476 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:38.343410 22476 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:38.343796 22017 ts_manager.cc:194] Registered new tserver with Master: 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585)
I20260925 15:00:38.344151 22017 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:59587
I20260925 15:00:38.347743 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:38.354820 22057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.349065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50326 (local address 127.8.13.254:42091)
0925 15:00:38.349156 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.349158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.349241 (+ 83us) server_negotiation.cc:407] Connection header received
0925 15:00:38.349309 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.349310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.349320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.349365 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.349941 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.350437 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.351164 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.351356 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.351412 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.352541 (+ 1129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.352543 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.352543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.352560 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.353729 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.354044 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.354045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.354046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.354103 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.354264 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.354264 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.354265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.354422 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.354517 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.354639 (+ 122us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.354727 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260925 15:00:38.356575 22017 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:00:38.358179 22017 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:00:38.365293 22491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.0.0.1:54404)
0925 15:00:38.360788 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.360797 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.360836 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.361350 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.361351 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.361353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:38.361408 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.361410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362795 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.362796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.364252 (+ 1456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.364256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.364935 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364936 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.365036 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.365051 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.365113 (+ 62us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.365172 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":57,"thread_start_us":102,"threads_started":1}
I20260925 15:00:38.365324 22488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:42585 (local address 127.0.0.1:53982)
0925 15:00:38.360578 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.360620 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.360689 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.361585 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.361586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.361588 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:38.361626 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.361627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362893 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.362894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.364048 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.364051 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.365020 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.365021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.365082 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.365086 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.365130 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.365175 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":113,"thread_start_us":54,"threads_started":1}
I20260925 15:00:38.365342 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.8.13.195:42585)
0925 15:00:38.361428 (+ 850us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.361432 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.361441 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:38.361478 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.361480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.361494 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.361567 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:38.361758 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362795 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364138 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.364854 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364926 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.364985 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.365226 (+ 241us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.365269 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":811,"thread_start_us":83,"threads_started":1}
I20260925 15:00:38.365669 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.8.13.193:37261)
0925 15:00:38.361042 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.361046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.361054 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:00:38.361089 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.361091 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.361107 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.361181 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:38.361436 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362706 (+ 1270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364348 (+ 1642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.365054 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.365119 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.365170 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.365547 (+ 377us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.365592 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":110,"threads_started":1}
I20260925 15:00:38.365801 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.0.0.1:45358)
0925 15:00:38.360790 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.360800 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.360849 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.361277 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.361280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.361283 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:38.361362 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.361363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362815 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.362816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.364273 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.364275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.365129 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.365130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.365182 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.365185 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.365230 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.365268 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":121,"thread_start_us":70,"threads_started":1}
I20260925 15:00:38.365938 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.360558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54404 (local address 127.8.13.194:37761)
0925 15:00:38.361064 (+ 506us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.361091 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.361102 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:00:38.361193 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.361196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.361212 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.361285 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:38.361484 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.362672 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364358 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.364855 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.364921 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.364984 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.365333 (+ 349us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.365808 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":80,"threads_started":1}
I20260925 15:00:38.366214 22410 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a (DEFAULT_TABLE table=test-table [id=6551779a1fbf47c8b34a62edcb787794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:38.366578 22410 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bed6bcc354d57a2344065f13eb29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:38.366479 22142 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a (DEFAULT_TABLE table=test-table [id=6551779a1fbf47c8b34a62edcb787794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:38.366631 22276 tablet_service.cc:1467] Processing CreateTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a (DEFAULT_TABLE table=test-table [id=6551779a1fbf47c8b34a62edcb787794]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:38.366796 22142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bed6bcc354d57a2344065f13eb29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:38.366904 22276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d5bed6bcc354d57a2344065f13eb29a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:38.369334 22493 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Bootstrap starting.
I20260925 15:00:38.370044 22494 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Bootstrap starting.
I20260925 15:00:38.370287 22493 tablet_bootstrap.cc:654] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:38.370712 22494 tablet_bootstrap.cc:654] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:38.370908 22493 log.cc:824] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.371250 22494 log.cc:824] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.371611 22495 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Bootstrap starting.
I20260925 15:00:38.371946 22494 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: No bootstrap required, opened a new log
I20260925 15:00:38.371999 22494 ts_tablet_manager.cc:1397] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:38.372186 22493 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: No bootstrap required, opened a new log
I20260925 15:00:38.372249 22495 tablet_bootstrap.cc:654] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:38.372315 22493 ts_tablet_manager.cc:1397] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:00:38.372803 22495 log.cc:824] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.373590 22495 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: No bootstrap required, opened a new log
I20260925 15:00:38.373670 22495 ts_tablet_manager.cc:1397] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:38.373618 22494 raft_consensus.cc:348] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.373801 22494 raft_consensus.cc:374] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:38.373844 22494 raft_consensus.cc:729] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5478a6a6a6c475fbab9b1ed0ffe9d4f, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.373956 22494 consensus_queue.cc:260] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.374214 22494 ts_tablet_manager.cc:1428] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:38.374336 22342 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
I20260925 15:00:38.374269 22493 raft_consensus.cc:348] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.374435 22493 raft_consensus.cc:374] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:38.374475 22493 raft_consensus.cc:729] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658742a535bb49f486a7fc9bc17f2c50, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.374583 22493 consensus_queue.cc:260] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.374843 22493 ts_tablet_manager.cc:1428] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:38.374958 22476 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
I20260925 15:00:38.376099 22495 raft_consensus.cc:348] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.376282 22495 raft_consensus.cc:374] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:38.376332 22495 raft_consensus.cc:729] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab472316eb694e3888f1ceb444891201, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.376475 22495 consensus_queue.cc:260] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.376709 22495 ts_tablet_manager.cc:1428] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:00:38.376746 22208 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
I20260925 15:00:38.388931 22500 raft_consensus.cc:482] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:38.389060 22500 raft_consensus.cc:504] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.389354 22500 leader_election.cc:288] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761), ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261)
I20260925 15:00:38.392233 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.389570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:39317 (local address 127.8.13.194:37761)
0925 15:00:38.389610 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.389612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.389622 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:00:38.389656 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.389657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.389669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.389728 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:38.389917 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.390448 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.391421 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.391865 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.391924 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.391933 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.392109 (+ 176us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.392146 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:00:38.392232 22478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.389503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.8.13.195:39317)
0925 15:00:38.389556 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.389568 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.389610 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.389779 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.389780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.389782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:38.389848 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.389851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.390537 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.390538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.391336 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.391339 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.391943 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.391944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.392031 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.392036 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.392104 (+ 68us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.392150 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:00:38.392388 22502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.389603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.8.13.195:48327)
0925 15:00:38.389782 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.389791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.389844 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.389957 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.389958 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.389960 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:38.390021 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.390022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.390691 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.390692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.391445 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.391448 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.392146 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.392147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.392218 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.392222 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.392277 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.392317 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":120,"threads_started":1}
I20260925 15:00:38.392594 22492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.389688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:48327 (local address 127.8.13.193:37261)
0925 15:00:38.389755 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.389757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.389845 (+ 88us) server_negotiation.cc:407] Connection header received
0925 15:00:38.389886 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.389887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.389900 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.389950 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:38.390116 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.390619 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.391578 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.392077 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.392143 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.392150 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.392484 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.392530 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260925 15:00:38.392623 22296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bed6bcc354d57a2344065f13eb29a" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" is_pre_election: true
I20260925 15:00:38.392779 22296 raft_consensus.cc:2454] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 0.
I20260925 15:00:38.393013 22364 leader_election.cc:302] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f; no voters:
I20260925 15:00:38.393126 22162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bed6bcc354d57a2344065f13eb29a" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab472316eb694e3888f1ceb444891201" is_pre_election: true
I20260925 15:00:38.393159 22500 raft_consensus.cc:2781] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:38.393222 22500 raft_consensus.cc:482] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:38.393245 22500 raft_consensus.cc:3037] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:38.393278 22162 raft_consensus.cc:2454] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 0.
I20260925 15:00:38.394008 22500 raft_consensus.cc:504] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.394122 22500 leader_election.cc:288] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 election: Requested vote from peers b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761), ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261)
I20260925 15:00:38.394344 22162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bed6bcc354d57a2344065f13eb29a" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab472316eb694e3888f1ceb444891201"
I20260925 15:00:38.394331 22296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d5bed6bcc354d57a2344065f13eb29a" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
I20260925 15:00:38.394425 22162 raft_consensus.cc:3037] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:38.394425 22296 raft_consensus.cc:3037] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:38.395354 22162 raft_consensus.cc:2454] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 1.
I20260925 15:00:38.395354 22296 raft_consensus.cc:2454] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 1.
I20260925 15:00:38.395601 22364 leader_election.cc:302] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658742a535bb49f486a7fc9bc17f2c50, ab472316eb694e3888f1ceb444891201; no voters:
I20260925 15:00:38.395710 22500 raft_consensus.cc:2781] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:38.395915 22500 raft_consensus.cc:686] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 1 LEADER]: Becoming Leader. State: Replica: 658742a535bb49f486a7fc9bc17f2c50, State: Running, Role: LEADER
I20260925 15:00:38.396028 22500 consensus_queue.cc:237] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.396944 22017 catalog_manager.cc:5495] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "658742a535bb49f486a7fc9bc17f2c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } health_report { overall_health: UNKNOWN } } }
I20260925 15:00:38.410161 22490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.407858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53994 (local address 127.8.13.195:42585)
0925 15:00:38.407900 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.407901 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.407910 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:38.407954 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.407955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.407975 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.408038 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:38.408234 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.408688 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.409449 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.409632 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.409673 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.409822 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:38.409922 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:38.409980 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.410059 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.410076 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:00:38.411224 22343 tablet.cc:2369] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:00:38.412082 22296 raft_consensus.cc:1261] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Refusing update from remote peer 658742a535bb49f486a7fc9bc17f2c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:38.412082 22162 raft_consensus.cc:1261] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Refusing update from remote peer 658742a535bb49f486a7fc9bc17f2c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:38.412431 22500 consensus_queue.cc:1035] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:38.412566 22503 consensus_queue.cc:1035] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:38.414986 22506 mvcc.cc:205] Tried to move back new op lower bound from 7333267203733151744 to 7333267203670949888. Current Snapshot: MvccSnapshot[applied={T|T < 7333267203733151744}]
I20260925 15:00:38.415357 22507 mvcc.cc:205] Tried to move back new op lower bound from 7333267203733151744 to 7333267203670949888. Current Snapshot: MvccSnapshot[applied={T|T < 7333267203733151744}]
I20260925 15:00:38.415731 22508 mvcc.cc:205] Tried to move back new op lower bound from 7333267203733151744 to 7333267203670949888. Current Snapshot: MvccSnapshot[applied={T|T < 7333267203733151744}]
I20260925 15:00:38.417380 22017 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:38.417465 22017 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:38.418202 22017 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Sending DeleteTablet for 3 replicas of tablet 1d5bed6bcc354d57a2344065f13eb29a
I20260925 15:00:38.418526 22410 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:38 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:53982
I20260925 15:00:38.418591 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22079
I20260925 15:00:38.418610 22276 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:38 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:54404
I20260925 15:00:38.419212 22515 tablet_replica.cc:330] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: stopping tablet replica
I20260925 15:00:38.419302 22515 raft_consensus.cc:2229] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:00:38.419389 22515 raft_consensus.cc:2258] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:38.419838 22515 ts_tablet_manager.cc:1905] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:38.421293 22515 ts_tablet_manager.cc:1918] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:38.421362 22515 log.cc:1196] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/1d5bed6bcc354d57a2344065f13eb29a
I20260925 15:00:38.421599 22515 ts_tablet_manager.cc:1939] T 1d5bed6bcc354d57a2344065f13eb29a P 658742a535bb49f486a7fc9bc17f2c50: Deleting consensus metadata
I20260925 15:00:38.421988 22003 catalog_manager.cc:4841] TS 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): tablet 1d5bed6bcc354d57a2344065f13eb29a (table test-table [id=6551779a1fbf47c8b34a62edcb787794]) successfully deleted
I20260925 15:00:38.424083 22516 tablet_replica.cc:330] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: stopping tablet replica
I20260925 15:00:38.424239 22516 raft_consensus.cc:2229] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260925 15:00:38.424332 22002 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:37261 (error 108)
I20260925 15:00:38.424373 22516 raft_consensus.cc:2258] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:38.424871 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.424745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.0.0.1:45366)
0925 15:00:38.424790 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.424835 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 15:00:38.424983 22002 catalog_manager.cc:4584] TS ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1d5bed6bcc354d57a2344065f13eb29a: Network error: Client connection negotiation failed: client connection to 127.8.13.193:37261: connect: Connection refused (error 111)
I20260925 15:00:38.425232 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22213
I20260925 15:00:38.425233 22516 ts_tablet_manager.cc:1905] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:38.430279 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22347
I20260925 15:00:38.430358 22487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.430231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.0.0.1:54414)
0925 15:00:38.430285 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.430321 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:37761: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 15:00:38.434824 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 21985
I20260925 15:00:38.439823 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--webserver_interface=127.8.13.254
--webserver_port=37107
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:38.520295 22518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.520532 22518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.520557 22518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.521855 22518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:38.521906 22518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.521920 22518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.521931 22518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:38.521942 22518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:38.523443 22518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=37107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.523694 22518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.523880 22518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.532148 22527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.532148 22525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.532148 22524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.532356 22518 server_base.cc:1034] running on GCE node
I20260925 15:00:38.532625 22518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.532945 22518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.534086 22518 hybrid_clock.cc:648] HybridClock initialized: now 1790348438534070 us; error 44 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.536695 22518 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:38.537199 22518 webserver.cc:460] Webserver started at http://127.8.13.254:37107/ using document root <none> and password file <none>
I20260925 15:00:38.537449 22518 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.537515 22518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.538955 22518 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:00:38.539647 22534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.539820 22518 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:38.539911 22518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "22538318a0654c13b0daac23579521b3"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "d63889d4e57167eb8c366211162dec44"
server_key_iv: "8431b84c39077d7555aa3e1e878edf31"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.540359 22518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.573164 22518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.575347 22518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.575492 22518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.579317 22518 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42091
I20260925 15:00:38.579428 22586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42091 every 8 connection(s)
I20260925 15:00:38.579775 22518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:38.581432 22587 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:00:38.582121 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap starting.
I20260925 15:00:38.584897 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22518
I20260925 15:00:38.585167 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:37261
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=35785
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:38.590369 22587 log.cc:824] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.592365 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:38.592667 22587 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap complete.
I20260925 15:00:38.594837 22587 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:38.595082 22587 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22538318a0654c13b0daac23579521b3, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.595237 22587 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:38.595311 22587 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:38.595366 22587 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:38.595414 22587 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:00:38.596513 22587 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:38.596627 22587 leader_election.cc:302] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22538318a0654c13b0daac23579521b3; no voters:
I20260925 15:00:38.596794 22587 leader_election.cc:288] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:00:38.596848 22591 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:00:38.596990 22591 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 LEADER]: Becoming Leader. State: Replica: 22538318a0654c13b0daac23579521b3, State: Running, Role: LEADER
I20260925 15:00:38.597072 22587 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:38.597152 22591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:38.597503 22593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22538318a0654c13b0daac23579521b3. Latest consensus state: current_term: 2 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:38.597509 22592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:38.597599 22593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:38.597618 22592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:38.598786 22607 catalog_manager.cc:1259] Loaded cluster ID: fb2b1016edae49a5916322f39693e4f5
I20260925 15:00:38.598825 22607 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: loading cluster ID for follower catalog manager: success
W20260925 15:00:38.599006 22607 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260925 15:00:38.599048 22607 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260925 15:00:38.599080 22599 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:38.599421 22599 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:38.599491 22599 catalog_manager.cc:1259] Loaded cluster ID: fb2b1016edae49a5916322f39693e4f5
I20260925 15:00:38.599522 22599 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:00:38.599689 22599 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260925 15:00:38.602355 22599 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260925 15:00:38.604336 22599 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:38.604547 22599 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Loaded TSK: 0
I20260925 15:00:38.604748 22599 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:00:38.677011 22589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.677238 22589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.677275 22589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.678550 22589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.678601 22589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.678639 22589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:38.680297 22589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:37261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=35785
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.680565 22589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.680786 22589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.685575 22614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.685583 22615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.685583 22617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.685984 22589 server_base.cc:1034] running on GCE node
I20260925 15:00:38.686201 22589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.686436 22589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.687633 22589 hybrid_clock.cc:648] HybridClock initialized: now 1790348438687608 us; error 36 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.690341 22589 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:38.690909 22589 webserver.cc:460] Webserver started at http://127.8.13.193:35785/ using document root <none> and password file <none>
I20260925 15:00:38.691169 22589 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.691255 22589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.693114 22589 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:38.693923 22624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.694124 22589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:38.694223 22589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ab472316eb694e3888f1ceb444891201"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "396edad3861e4fe823421e33bba471d8"
server_key_iv: "f22732c657ffaf2e6f7e00df51ed5deb"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.694691 22589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.738340 22589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.741016 22589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.741168 22589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.741400 22589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:38.741876 22631 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:00:38.743496 22589 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:00:38.743583 22589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:00:38.743618 22589 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:00:38.744524 22589 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:00:38.744586 22589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:38.744633 22631 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Bootstrap starting.
I20260925 15:00:38.751653 22589 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:37261
I20260925 15:00:38.751766 22738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:37261 every 8 connection(s)
I20260925 15:00:38.752122 22589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:38.754544 22631 log.cc:824] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.756735 22631 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:38.757002 22631 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Bootstrap complete.
I20260925 15:00:38.757766 22631 ts_tablet_manager.cc:1397] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.006s
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:38.758816 22631 raft_consensus.cc:348] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.759086 22631 raft_consensus.cc:729] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab472316eb694e3888f1ceb444891201, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.759234 22631 consensus_queue.cc:260] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.759502 22631 ts_tablet_manager.cc:1428] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260925 15:00:38.760929 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22589
I20260925 15:00:38.761224 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:37761
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=34175
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:38.763427 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.752961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43547 (local address 127.8.13.254:42091)
0925 15:00:38.753146 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.753156 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.753199 (+ 43us) server_negotiation.cc:407] Connection header received
0925 15:00:38.753809 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.753814 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.753864 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.753987 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.754654 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.755643 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.756667 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.757206 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.757303 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.759737 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.759742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.759742 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.759754 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.761655 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.762099 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.762102 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.762103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.762230 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.762506 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.762507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.762507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.762753 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.762818 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.763156 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.763273 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
I20260925 15:00:38.763502 22741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.752833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:43547)
0925 15:00:38.752999 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.753039 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.753574 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.754031 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.754035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.754330 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.754531 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.754536 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.755760 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.755761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.756575 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.756577 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.757319 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.757320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.757446 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.758000 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.758008 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.759602 (+ 1594us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.761761 (+ 2159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.761769 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.761771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.762015 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.762290 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.762290 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.762291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.762437 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.762815 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.762821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.762993 (+ 172us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.763106 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.763302 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
I20260925 15:00:38.764257 22739 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:38.764353 22739 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:38.764532 22739 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:38.765339 22551 ts_manager.cc:194] Registered new tserver with Master: ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261)
I20260925 15:00:38.766569 22551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:43547
I20260925 15:00:38.767591 22739 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 15:00:38.773984 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.765982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.0.0.1:45370)
0925 15:00:38.766223 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.766249 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.766303 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.766550 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.766556 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.766854 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.766924 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.766927 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.767994 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.767995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.768626 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.768629 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.769067 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.769068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.769148 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.769670 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.769675 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.771270 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.772896 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.772903 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.772906 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.773104 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.773290 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.773291 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.773291 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.773354 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.773612 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.773619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.773676 (+ 57us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.773749 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.773879 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":114,"threads_started":1}
I20260925 15:00:38.773984 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.766044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45370 (local address 127.8.13.193:37261)
0925 15:00:38.766394 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.766398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.766405 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:00:38.766430 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.766431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.766442 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.766496 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.766985 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.767922 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.768701 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.768993 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.769058 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.771353 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.771360 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.771361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.771367 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.772818 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.773177 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.773180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.773180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.773233 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.773403 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.773404 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.773404 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.773551 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.773621 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.773791 (+ 170us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.773882 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":59,"threads_started":1}
I20260925 15:00:38.774375 22673 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:38 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:45370
I20260925 15:00:38.774870 22748 tablet_replica.cc:330] stopping tablet replica
I20260925 15:00:38.774981 22748 raft_consensus.cc:2229] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:38.775054 22748 raft_consensus.cc:2258] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:38.775360 22748 ts_tablet_manager.cc:1905] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:38.777645 22748 ts_tablet_manager.cc:1918] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:38.777719 22748 log.cc:1196] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/1d5bed6bcc354d57a2344065f13eb29a
I20260925 15:00:38.777920 22748 ts_tablet_manager.cc:1939] T 1d5bed6bcc354d57a2344065f13eb29a P ab472316eb694e3888f1ceb444891201: Deleting consensus metadata
I20260925 15:00:38.778465 22536 catalog_manager.cc:4841] TS ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261): tablet 1d5bed6bcc354d57a2344065f13eb29a (table test-table [id=6551779a1fbf47c8b34a62edcb787794]) successfully deleted
W20260925 15:00:38.846634 22745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.846868 22745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.846896 22745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.848402 22745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.848469 22745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.848505 22745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:38.850353 22745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:37761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=34175
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.850589 22745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.850785 22745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.855873 22754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.855865 22756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.855947 22745 server_base.cc:1034] running on GCE node
W20260925 15:00:38.855860 22753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.856408 22745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.856669 22745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.857839 22745 hybrid_clock.cc:648] HybridClock initialized: now 1790348438857827 us; error 40 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.860364 22745 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:38.860811 22745 webserver.cc:460] Webserver started at http://127.8.13.194:34175/ using document root <none> and password file <none>
I20260925 15:00:38.861016 22745 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.861068 22745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.862406 22745 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:00:38.863070 22763 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.863245 22745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:00:38.863332 22745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "b96f4a705297e623861e45b01170c0dd"
server_key_iv: "9cb296dc27dc4dc28bfebcac9c52d110"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.863714 22745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:38.874560 22745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:38.876449 22745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:38.876565 22745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:38.876750 22745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:38.877204 22770 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:00:38.878816 22745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:00:38.878877 22745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:00:38.878906 22745 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:00:38.879637 22745 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:00:38.879678 22745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:38.879762 22770 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Bootstrap starting.
I20260925 15:00:38.881524 22770 log.cc:824] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:38.884615 22770 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:38.884918 22770 tablet_bootstrap.cc:492] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Bootstrap complete.
I20260925 15:00:38.885066 22770 ts_tablet_manager.cc:1397] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260925 15:00:38.885382 22745 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:37761
I20260925 15:00:38.885457 22878 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:37761 every 8 connection(s)
I20260925 15:00:38.885733 22745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:38.886778 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22745
I20260925 15:00:38.886756 22770 raft_consensus.cc:348] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.887060 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:42585
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=37127
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:38.887095 22770 raft_consensus.cc:729] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5478a6a6a6c475fbab9b1ed0ffe9d4f, State: Initialized, Role: FOLLOWER
I20260925 15:00:38.887248 22770 consensus_queue.cc:260] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } } peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } }
I20260925 15:00:38.887564 22770 ts_tablet_manager.cc:1428] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:38.895833 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.886439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:47347 (local address 127.8.13.254:42091)
0925 15:00:38.886500 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.886502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.886677 (+ 175us) server_negotiation.cc:407] Connection header received
0925 15:00:38.886842 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.886843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.886855 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.886907 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.887786 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.888300 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.889289 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.889503 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.889560 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.892495 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.892496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.892497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.892505 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.893988 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.894636 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.894637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.894637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.894705 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.895027 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.895028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.895028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.895160 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.895216 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.895663 (+ 447us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.895759 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:00:38.896076 22881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.886362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:47347)
0925 15:00:38.886544 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.886587 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.886766 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.887014 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.887018 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.887370 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.887685 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.887692 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.888407 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.888408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.889201 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.889203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.889578 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.889579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.889719 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.890269 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.890278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.892377 (+ 2099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.894104 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.894116 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.894119 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.894477 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.894784 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.894785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.894785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.894953 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.895228 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.895236 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.895438 (+ 202us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.895607 (+ 169us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.895821 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":84,"threads_started":1}
I20260925 15:00:38.896643 22879 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:38.896764 22879 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:38.896953 22879 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:38.897534 22551 ts_manager.cc:194] Registered new tserver with Master: b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
I20260925 15:00:38.898176 22551 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:47347
I20260925 15:00:38.899072 22879 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.194@KRBTEST.COM
I20260925 15:00:38.906065 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.897854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.0.0.1:54422)
0925 15:00:38.897903 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:38.897914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:38.897951 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:38.898250 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:38.898251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:38.898428 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:38.898484 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.898486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.899754 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.899755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.900601 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:38.900604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.901082 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.901083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:38.901147 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.901577 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:38.901582 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:38.902872 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:38.904907 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.904908 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.904908 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.905082 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.905328 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:38.905329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:38.905329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:38.905402 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:38.905788 (+ 386us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:38.905789 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:38.905838 (+ 49us) client_negotiation.cc:769] Sending connection context
0925 15:00:38.905901 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:00:38.905991 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:00:38.906153 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:38.897924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54422 (local address 127.8.13.194:37761)
0925 15:00:38.898100 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:38.898105 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:38.898114 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:38.898148 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:38.898151 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:38.898165 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:38.898461 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:38.898575 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.899692 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.900692 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:38.901019 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:38.901078 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:38.902954 (+ 1876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:38.902964 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:38.902964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:38.902972 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:38.904809 (+ 1837us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.905165 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.905169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.905169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.905234 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:38.905467 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:38.905467 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:38.905468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:38.905703 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:38.905782 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:38.905942 (+ 160us) server_negotiation.cc:299] Negotiation successful
0925 15:00:38.906041 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260925 15:00:38.906505 22813 tablet_service.cc:1514] Processing DeleteTablet for tablet 1d5bed6bcc354d57a2344065f13eb29a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:38 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:54422
I20260925 15:00:38.907063 22885 tablet_replica.cc:330] stopping tablet replica
I20260925 15:00:38.907157 22885 raft_consensus.cc:2229] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:38.907231 22885 raft_consensus.cc:2258] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:38.907588 22885 ts_tablet_manager.cc:1905] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:38.909565 22885 ts_tablet_manager.cc:1918] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:38.909642 22885 log.cc:1196] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/1d5bed6bcc354d57a2344065f13eb29a
I20260925 15:00:38.909848 22885 ts_tablet_manager.cc:1939] T 1d5bed6bcc354d57a2344065f13eb29a P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting consensus metadata
I20260925 15:00:38.910347 22536 catalog_manager.cc:4841] TS b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): tablet 1d5bed6bcc354d57a2344065f13eb29a (table test-table [id=6551779a1fbf47c8b34a62edcb787794]) successfully deleted
W20260925 15:00:38.973100 22883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:38.973330 22883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:38.973369 22883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:38.974610 22883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:38.974664 22883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:38.974702 22883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:38.976281 22883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:42585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=37127
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22883
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:38.976547 22883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:38.976774 22883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:38.981685 22891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.981781 22883 server_base.cc:1034] running on GCE node
W20260925 15:00:38.981703 22893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:38.981685 22890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:38.982141 22883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:38.982385 22883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:38.983542 22883 hybrid_clock.cc:648] HybridClock initialized: now 1790348438983512 us; error 30 us; skew 500 ppm
Sep 25 15:00:38 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:38.986110 22883 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:38.986557 22883 webserver.cc:460] Webserver started at http://127.8.13.195:37127/ using document root <none> and password file <none>
I20260925 15:00:38.986774 22883 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:38.986840 22883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:38.988296 22883 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:38.988883 22900 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:38.989056 22883 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:38.989140 22883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "658742a535bb49f486a7fc9bc17f2c50"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "944d27d2081e5f9df4806e0f3a9a97d5"
server_key_iv: "19feabef80bcc910060a59fa575a353e"
server_key_version: "encryptionkey@0"
I20260925 15:00:38.989496 22883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:39.000609 22883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:39.002779 22883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:39.002918 22883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:39.003131 22883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:39.003427 22883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:39.003458 22883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.003487 22883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:39.003512 22883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.009563 22883 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:42585
I20260925 15:00:39.009629 23013 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:42585 every 8 connection(s)
I20260925 15:00:39.009972 22883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:39.012028 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 22883
Sep 25 15:00:39 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:39.019311 23018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.012423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50340 (local address 127.8.13.254:42091)
0925 15:00:39.012546 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.012548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.012557 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:39.012634 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.012635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.012647 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.012698 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:39.013315 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.013884 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.014662 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.014849 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.014904 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.016352 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:39.016354 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:39.016354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:39.016362 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:39.018016 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:39.018490 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:39.018491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:39.018491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:39.018543 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:39.018726 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:39.018726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:39.018727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:39.018848 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:39.019002 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.019120 (+ 118us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.019194 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260925 15:00:39.019908 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.010780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:41349 (local address 127.8.13.254:42091)
0925 15:00:39.010843 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.010845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.010998 (+ 153us) server_negotiation.cc:407] Connection header received
0925 15:00:39.011506 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.011507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.011517 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.011606 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:39.012758 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.013346 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.014348 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.014553 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.014610 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.016928 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:39.016929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:39.016929 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:39.016936 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:39.018334 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:39.018837 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:39.018838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:39.018838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:39.018890 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:39.019224 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:39.019225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:39.019226 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:39.019340 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:39.019407 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.019751 (+ 344us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.019826 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:00:39.020159 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.010722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:41349)
0925 15:00:39.010883 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.010927 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:39.011412 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:39.011682 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:39.011686 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:39.012417 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:39.012667 (+ 250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.012673 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.013440 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.013441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.014267 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.014270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.014627 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.014628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.014746 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.015291 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:39.015303 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:39.016823 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:39.018425 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:39.018435 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:39.018438 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:39.018771 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:39.018983 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:39.018985 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:39.018985 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:39.019159 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:39.019417 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:39.019438 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:39.019632 (+ 194us) client_negotiation.cc:769] Sending connection context
0925 15:00:39.019724 (+ 92us) client_negotiation.cc:240] Negotiation successful
0925 15:00:39.019903 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260925 15:00:39.020646 23014 heartbeater.cc:347] Connected to a master server at 127.8.13.254:42091
I20260925 15:00:39.020752 23014 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:39.020948 23014 heartbeater.cc:511] Master 127.8.13.254:42091 requested a full tablet report, sending...
I20260925 15:00:39.021340 22550 ts_manager.cc:194] Registered new tserver with Master: 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585)
I20260925 15:00:39.021780 22550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:41349
I20260925 15:00:39.031886 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:39.034909 22742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.032317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.8.13.254:42091)
0925 15:00:39.032373 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.032375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.032461 (+ 86us) server_negotiation.cc:407] Connection header received
0925 15:00:39.032573 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.032574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.032612 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.032673 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:39.032800 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.033293 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.034097 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.034296 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.034351 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.034541 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:39.034655 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:39.034741 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.034813 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.034835 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:00:39.035207 22550 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50344:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:00:39.037047 22550 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:00:39.039145 22673 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 (DEFAULT_TABLE table=test-table [id=6d6e4ec11f1f4733938781104a9738e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:39.039191 22813 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 (DEFAULT_TABLE table=test-table [id=6d6e4ec11f1f4733938781104a9738e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:39.039345 22673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4579e2019e4bbe876ca696df47ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:39.039374 22813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4579e2019e4bbe876ca696df47ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:39.040792 22770 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Bootstrap starting.
I20260925 15:00:39.041260 22770 tablet_bootstrap.cc:654] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:39.041600 22631 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Bootstrap starting.
I20260925 15:00:39.041918 22770 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: No bootstrap required, opened a new log
I20260925 15:00:39.041926 22631 tablet_bootstrap.cc:654] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:39.041968 22770 ts_tablet_manager.cc:1397] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.042286 22770 raft_consensus.cc:348] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.042364 22770 raft_consensus.cc:374] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:39.042413 22770 raft_consensus.cc:729] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b5478a6a6a6c475fbab9b1ed0ffe9d4f, State: Initialized, Role: FOLLOWER
I20260925 15:00:39.042477 22770 consensus_queue.cc:260] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.042577 22770 ts_tablet_manager.cc:1428] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:39.043745 22631 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: No bootstrap required, opened a new log
I20260925 15:00:39.043797 22631 ts_tablet_manager.cc:1397] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:00:39.044112 22631 raft_consensus.cc:348] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.044183 22631 raft_consensus.cc:374] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:39.044252 22631 raft_consensus.cc:729] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab472316eb694e3888f1ceb444891201, State: Initialized, Role: FOLLOWER
I20260925 15:00:39.044305 22631 consensus_queue.cc:260] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.044417 22631 ts_tablet_manager.cc:1428] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.044588 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.038945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:42585 (local address 127.0.0.1:54006)
0925 15:00:39.038990 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.039001 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:39.039044 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:39.041179 (+ 2135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:39.041180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:39.041182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:39.041229 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.041231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.042325 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.042326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.043457 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.043460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.044338 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.044339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.044424 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.044451 (+ 27us) client_negotiation.cc:769] Sending connection context
0925 15:00:39.044482 (+ 31us) client_negotiation.cc:240] Negotiation successful
0925 15:00:39.044527 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:00:39.044920 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.040828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.8.13.195:42585)
0925 15:00:39.041025 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.041030 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.041039 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:39.041073 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.041074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.041088 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.041163 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:39.041283 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.042234 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.043591 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.044240 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.044694 (+ 454us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.044773 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.044801 (+ 28us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.044842 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260925 15:00:39.045567 22948 tablet_service.cc:1467] Processing CreateTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 (DEFAULT_TABLE table=test-table [id=6d6e4ec11f1f4733938781104a9738e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:39.045850 22948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3c4579e2019e4bbe876ca696df47ba85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:39.048676 23023 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Bootstrap starting.
I20260925 15:00:39.049355 23023 tablet_bootstrap.cc:654] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:39.049821 23023 log.cc:824] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:39.050500 23023 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: No bootstrap required, opened a new log
I20260925 15:00:39.050571 23023 ts_tablet_manager.cc:1397] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:00:39.052126 23023 raft_consensus.cc:348] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.052278 23023 raft_consensus.cc:374] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:39.052302 23023 raft_consensus.cc:729] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 658742a535bb49f486a7fc9bc17f2c50, State: Initialized, Role: FOLLOWER
I20260925 15:00:39.052398 23023 consensus_queue.cc:260] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.052693 23023 ts_tablet_manager.cc:1428] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:39.052716 23014 heartbeater.cc:503] Master 127.8.13.254:42091 was elected leader, sending a full tablet report...
I20260925 15:00:39.071825 23025 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:39.071985 23025 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.072347 23025 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
I20260925 15:00:39.075639 23016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.072528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.8.13.195:42447)
0925 15:00:39.072575 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.072587 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:39.072634 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:39.072825 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:39.072826 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:39.072828 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:39.072906 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.072908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.073651 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.073652 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.074532 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.074535 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.075329 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.075330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.075414 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.075419 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:39.075478 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:00:39.075523 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:00:39.075665 22747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.072610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:42447 (local address 127.8.13.193:37261)
0925 15:00:39.072664 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.072666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.072677 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:00:39.072707 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.072708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.072735 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.072822 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:39.072976 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.073555 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.074648 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.075255 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.075321 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.075385 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.075513 (+ 128us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.075588 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:00:39.075841 23026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.072640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.8.13.195:32807)
0925 15:00:39.072800 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.072810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:39.072867 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:39.073018 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:39.073019 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:39.073020 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:39.073101 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.073103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.073955 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.073956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.074827 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:39.074830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.075573 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.075574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:39.075645 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.075650 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:39.075723 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:00:39.075764 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":109,"threads_started":1}
I20260925 15:00:39.075857 22884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.072748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:32807 (local address 127.8.13.194:37761)
0925 15:00:39.072794 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.072796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.072868 (+ 72us) server_negotiation.cc:407] Connection header received
0925 15:00:39.072927 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.072928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.072953 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.073007 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:39.073198 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.073849 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.074927 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.075475 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.075568 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.075648 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.075731 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.075778 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:00:39.075961 22692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4579e2019e4bbe876ca696df47ba85" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab472316eb694e3888f1ceb444891201" is_pre_election: true
I20260925 15:00:39.076112 22692 raft_consensus.cc:2454] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 0.
I20260925 15:00:39.076346 22902 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658742a535bb49f486a7fc9bc17f2c50, ab472316eb694e3888f1ceb444891201; no voters:
I20260925 15:00:39.076496 23025 raft_consensus.cc:2781] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:39.076452 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4579e2019e4bbe876ca696df47ba85" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" is_pre_election: true
I20260925 15:00:39.076545 23025 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:39.076568 23025 raft_consensus.cc:3037] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:39.076608 22817 raft_consensus.cc:2454] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 0.
I20260925 15:00:39.077322 23025 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.077435 23025 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 election: Requested vote from peers ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
I20260925 15:00:39.077610 22817 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4579e2019e4bbe876ca696df47ba85" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
I20260925 15:00:39.077625 22692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3c4579e2019e4bbe876ca696df47ba85" candidate_uuid: "658742a535bb49f486a7fc9bc17f2c50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab472316eb694e3888f1ceb444891201"
I20260925 15:00:39.077687 22817 raft_consensus.cc:3037] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:39.077711 22692 raft_consensus.cc:3037] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:39.078346 22817 raft_consensus.cc:2454] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 1.
I20260925 15:00:39.078452 22692 raft_consensus.cc:2454] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 658742a535bb49f486a7fc9bc17f2c50 in term 1.
I20260925 15:00:39.078516 22902 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f; no voters:
I20260925 15:00:39.078665 23025 raft_consensus.cc:2781] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:39.078811 23025 raft_consensus.cc:686] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 1 LEADER]: Becoming Leader. State: Replica: 658742a535bb49f486a7fc9bc17f2c50, State: Running, Role: LEADER
I20260925 15:00:39.078899 23025 consensus_queue.cc:237] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.079872 22550 catalog_manager.cc:5495] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "658742a535bb49f486a7fc9bc17f2c50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } health_report { overall_health: UNKNOWN } } }
I20260925 15:00:39.089210 23020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.086844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54012 (local address 127.8.13.195:42585)
0925 15:00:39.086901 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:39.086903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:39.086911 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:00:39.086962 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:39.086963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:39.086983 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:39.087048 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:39.087170 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.087711 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.088412 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:39.088656 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:39.088716 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:39.088854 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:39.088969 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:39.089043 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:39.089080 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:00:39.089102 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:00:39.091117 22817 raft_consensus.cc:1261] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Refusing update from remote peer 658742a535bb49f486a7fc9bc17f2c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:39.091117 22692 raft_consensus.cc:1261] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Refusing update from remote peer 658742a535bb49f486a7fc9bc17f2c50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:39.091470 23025 consensus_queue.cc:1035] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:39.091672 23027 consensus_queue.cc:1035] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:39.094004 23030 mvcc.cc:205] Tried to move back new op lower bound from 7333267206514450432 to 7333267206467928064. Current Snapshot: MvccSnapshot[applied={T|T < 7333267206514450432}]
I20260925 15:00:39.094182 23031 mvcc.cc:205] Tried to move back new op lower bound from 7333267206514450432 to 7333267206467928064. Current Snapshot: MvccSnapshot[applied={T|T < 7333267206514450432}]
I20260925 15:00:39.096159 22551 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:39.096269 22551 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:50344:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:39.097071 22551 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Sending DeleteTablet for 3 replicas of tablet 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:39.097405 22673 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:39 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:45370
I20260925 15:00:39.097429 22948 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:39 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:54006
I20260925 15:00:39.097460 22813 tablet_service.cc:1514] Processing DeleteTablet for tablet 3c4579e2019e4bbe876ca696df47ba85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:39 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:54422
I20260925 15:00:39.097532 22748 tablet_replica.cc:330] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: stopping tablet replica
I20260925 15:00:39.097543 22885 tablet_replica.cc:330] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: stopping tablet replica
I20260925 15:00:39.097587 22885 raft_consensus.cc:2229] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:39.097582 22748 raft_consensus.cc:2229] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:39.097627 22885 raft_consensus.cc:2258] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:39.097667 22748 raft_consensus.cc:2258] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:39.097889 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22589
I20260925 15:00:39.098021 22885 ts_tablet_manager.cc:1905] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:39.099681 22885 ts_tablet_manager.cc:1918] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:39.099741 22885 log.cc:1196] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:39.099932 22885 ts_tablet_manager.cc:1939] T 3c4579e2019e4bbe876ca696df47ba85 P b5478a6a6a6c475fbab9b1ed0ffe9d4f: Deleting consensus metadata
I20260925 15:00:39.100363 22536 catalog_manager.cc:4841] TS b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): tablet 3c4579e2019e4bbe876ca696df47ba85 (table test-table [id=6d6e4ec11f1f4733938781104a9738e3]) successfully deleted
I20260925 15:00:39.103763 23038 tablet_replica.cc:330] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: stopping tablet replica
I20260925 15:00:39.103927 23038 raft_consensus.cc:2229] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:00:39.104102 23038 raft_consensus.cc:2258] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260925 15:00:39.104557 22536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:37261 (error 108)
I20260925 15:00:39.104812 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22745
I20260925 15:00:39.105399 23038 ts_tablet_manager.cc:1905] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:39.107051 23038 ts_tablet_manager.cc:1918] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:39.107116 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.106972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:37261 (local address 127.0.0.1:45380)
0925 15:00:39.107030 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.107079 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:37261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260925 15:00:39.107154 23038 log.cc:1196] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:39.107414 23038 ts_tablet_manager.cc:1939] T 3c4579e2019e4bbe876ca696df47ba85 P 658742a535bb49f486a7fc9bc17f2c50: Deleting consensus metadata
W20260925 15:00:39.107236 22536 catalog_manager.cc:4584] TS ab472316eb694e3888f1ceb444891201 (127.8.13.193:37261): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3c4579e2019e4bbe876ca696df47ba85: Network error: Client connection negotiation failed: client connection to 127.8.13.193:37261: connect: Connection refused (error 111)
I20260925 15:00:39.107858 22537 catalog_manager.cc:4841] TS 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): tablet 3c4579e2019e4bbe876ca696df47ba85 (table test-table [id=6d6e4ec11f1f4733938781104a9738e3]) successfully deleted
I20260925 15:00:39.112339 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22883
I20260925 15:00:39.116677 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 22518
I20260925 15:00:39.121192 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--webserver_interface=127.8.13.254
--webserver_port=37107
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42091
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:39.201638 23040 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:39.201927 23040 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:39.201956 23040 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:39.203352 23040 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:39.203496 23040 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:39.203621 23040 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:39.203644 23040 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:39.203660 23040 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:39.205497 23040 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42091
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=37107
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23040
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:39.205785 23040 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:39.206018 23040 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:39.210915 23045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.210927 23048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.210927 23046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.211133 23040 server_base.cc:1034] running on GCE node
I20260925 15:00:39.211447 23040 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:39.211869 23040 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:39.213080 23040 hybrid_clock.cc:648] HybridClock initialized: now 1790348439213059 us; error 40 us; skew 500 ppm
Sep 25 15:00:39 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:39.219424 23040 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:39.219983 23040 webserver.cc:460] Webserver started at http://127.8.13.254:37107/ using document root <none> and password file <none>
I20260925 15:00:39.220240 23040 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:39.220297 23040 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:39.221801 23040 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.222599 23055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.222805 23040 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.222906 23040 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "22538318a0654c13b0daac23579521b3"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "d63889d4e57167eb8c366211162dec44"
server_key_iv: "8431b84c39077d7555aa3e1e878edf31"
server_key_version: "encryptionkey@0"
I20260925 15:00:39.223282 23040 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:39.231941 23040 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:39.233937 23040 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:39.234097 23040 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:39.238194 23040 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42091
I20260925 15:00:39.238241 23107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42091 every 8 connection(s)
I20260925 15:00:39.238626 23040 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:39.240299 23108 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:00:39.240947 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap starting.
I20260925 15:00:39.246078 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23040
I20260925 15:00:39.246435 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:37261
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=35785
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:39.249650 23108 log.cc:824] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:39.252300 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:39.252596 23108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3: Bootstrap complete.
I20260925 15:00:39.254710 23108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:39.254949 23108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22538318a0654c13b0daac23579521b3, State: Initialized, Role: FOLLOWER
I20260925 15:00:39.255095 23108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:39.255162 23108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:39.255211 23108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:39.255266 23108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:00:39.256385 23108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:39.256489 23108 leader_election.cc:302] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22538318a0654c13b0daac23579521b3; no voters:
I20260925 15:00:39.256676 23108 leader_election.cc:288] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:00:39.256732 23112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:00:39.256847 23112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [term 3 LEADER]: Becoming Leader. State: Replica: 22538318a0654c13b0daac23579521b3, State: Running, Role: LEADER
I20260925 15:00:39.256944 23112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } }
I20260925 15:00:39.256984 23108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:39.257273 23114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:39.257308 23112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22538318a0654c13b0daac23579521b3. Latest consensus state: current_term: 3 leader_uuid: "22538318a0654c13b0daac23579521b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22538318a0654c13b0daac23579521b3" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42091 } } }
I20260925 15:00:39.257491 23112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:39.257468 23114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22538318a0654c13b0daac23579521b3 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:39.257925 23122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:39.258802 23122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:39.258916 23122 catalog_manager.cc:1259] Loaded cluster ID: fb2b1016edae49a5916322f39693e4f5
I20260925 15:00:39.258949 23122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260925 15:00:39.259181 23122 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7feca08889ad google::LogMessage::Fail() at ??:0
@ 0x7feca088b88a google::LogMessage::SendToLog() at ??:0
@ 0x7feca08884de google::LogMessage::Flush() at ??:0
@ 0x7feca0889a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7feca87a90f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7feca87b28b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7feca27560d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7feca274b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7feca35af6db start_thread at ??:0
@ 0x7fec9f38661f clone at ??:0
W20260925 15:00:39.342213 23110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:39.342422 23110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:39.342445 23110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:39.343693 23110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:39.343735 23110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:39.343761 23110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:39.345256 23110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:37261
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=35785
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:39.345487 23110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:39.345726 23110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:39.350502 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.350590 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.350514 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.351236 23110 server_base.cc:1034] running on GCE node
I20260925 15:00:39.351461 23110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:39.351845 23110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:39.353035 23110 hybrid_clock.cc:648] HybridClock initialized: now 1790348439353020 us; error 47 us; skew 500 ppm
Sep 25 15:00:39 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:39.355804 23110 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:39.356261 23110 webserver.cc:460] Webserver started at http://127.8.13.193:35785/ using document root <none> and password file <none>
I20260925 15:00:39.356473 23110 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:39.356523 23110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:39.357996 23110 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.358739 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.358894 23110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:39.358982 23110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "ab472316eb694e3888f1ceb444891201"
format_stamp: "Formatted at 2026-09-25 15:00:37 on dist-test-slave-15hz"
server_key: "396edad3861e4fe823421e33bba471d8"
server_key_iv: "f22732c657ffaf2e6f7e00df51ed5deb"
server_key_version: "encryptionkey@0"
I20260925 15:00:39.359349 23110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:39.386581 23110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:39.388978 23110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:39.389142 23110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:39.389418 23110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:39.389951 23161 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:00:39.391438 23110 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:00:39.391510 23110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:00:39.391571 23110 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:00:39.392483 23110 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:00:39.392531 23110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:39.392596 23161 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Bootstrap starting.
I20260925 15:00:39.399850 23110 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:37261
I20260925 15:00:39.399945 23268 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:37261 every 8 connection(s)
I20260925 15:00:39.400342 23110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:39.402441 23161 log.cc:824] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:39.402730 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23110
I20260925 15:00:39.402981 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:37761
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=34175
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:00:39.405143 23161 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:00:39.405407 23161 tablet_bootstrap.cc:492] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Bootstrap complete.
I20260925 15:00:39.406128 23161 ts_tablet_manager.cc:1397] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.013s
I20260925 15:00:39.407516 23161 raft_consensus.cc:348] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.407903 23161 raft_consensus.cc:729] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab472316eb694e3888f1ceb444891201, State: Initialized, Role: FOLLOWER
I20260925 15:00:39.408084 23161 consensus_queue.cc:260] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:39.408380 23161 ts_tablet_manager.cc:1428] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
W20260925 15:00:39.487257 23273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:39.487504 23273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:39.487555 23273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:39.488873 23273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:39.488930 23273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:39.488960 23273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:39.490595 23273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:37761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=34175
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23273
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:39.490869 23273 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:39.491111 23273 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:39.496330 23279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.496362 23280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.496613 23273 server_base.cc:1034] running on GCE node
W20260925 15:00:39.496330 23282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.496973 23273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:39.497251 23273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:39.498399 23273 hybrid_clock.cc:648] HybridClock initialized: now 1790348439498381 us; error 39 us; skew 500 ppm
Sep 25 15:00:39 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:39.501324 23273 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:39.501785 23273 webserver.cc:460] Webserver started at http://127.8.13.194:34175/ using document root <none> and password file <none>
I20260925 15:00:39.501993 23273 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:39.502046 23273 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:39.503502 23273 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:00:39.504243 23289 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.504408 23273 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
W20260925 15:00:39.504575 23271 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:00:39.401084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:42329)
0925 15:00:39.401329 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.401383 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:39.402007 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:39.504362 (+102355us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":128,"threads_started":1}
W20260925 15:00:39.504797 23269 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:42091 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:42091: Client connection negotiation failed: client connection to 127.8.13.254:42091: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:00:39.504515 23273 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "b96f4a705297e623861e45b01170c0dd"
server_key_iv: "9cb296dc27dc4dc28bfebcac9c52d110"
server_key_version: "encryptionkey@0"
I20260925 15:00:39.504978 23273 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:39.505151 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.505021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:49179)
0925 15:00:39.505065 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.505122 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260925 15:00:39.505520 23271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.505383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:56603)
0925 15:00:39.505457 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.505493 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260925 15:00:39.505663 23269 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:00:39.523089 23273 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:39.525286 23273 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:39.525403 23273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:39.525593 23273 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:39.525898 23273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:39.525933 23273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.525954 23273 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:39.525964 23273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.531702 23273 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:37761
I20260925 15:00:39.531759 23402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:37761 every 8 connection(s)
I20260925 15:00:39.532099 23273 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:39.533161 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.532797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:36365)
0925 15:00:39.532947 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.533018 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":72,"threads_started":1}
W20260925 15:00:39.533521 23403 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:42091 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:42091: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
I20260925 15:00:39.533806 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.533681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:51447)
0925 15:00:39.533749 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.533781 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260925 15:00:39.534124 23405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.534066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:52567)
0925 15:00:39.534090 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.534110 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260925 15:00:39.534211 23403 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:00:39.540086 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23273
I20260925 15:00:39.540342 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:42585
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:42091
--webserver_port=37127
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:39.622359 23406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:39.622591 23406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:39.622627 23406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:39.623900 23406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:39.623946 23406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:39.623983 23406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:39.625578 23406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:42585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=37127
--tserver_master_addrs=127.8.13.254:42091
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:39.625854 23406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:39.626087 23406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:39.631129 23412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:39.631207 23411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.631289 23406 server_base.cc:1034] running on GCE node
W20260925 15:00:39.631119 23414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:39.631616 23406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:39.631886 23406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:39.633050 23406 hybrid_clock.cc:648] HybridClock initialized: now 1790348439633016 us; error 36 us; skew 500 ppm
Sep 25 15:00:39 dist-test-slave-15hz krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:39.635618 23406 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:39.636026 23406 webserver.cc:460] Webserver started at http://127.8.13.195:37127/ using document root <none> and password file <none>
I20260925 15:00:39.636224 23406 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:39.636265 23406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:39.637730 23406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:39.638343 23421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.638497 23406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:39.638578 23406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "658742a535bb49f486a7fc9bc17f2c50"
format_stamp: "Formatted at 2026-09-25 15:00:38 on dist-test-slave-15hz"
server_key: "944d27d2081e5f9df4806e0f3a9a97d5"
server_key_iv: "19feabef80bcc910060a59fa575a353e"
server_key_version: "encryptionkey@0"
I20260925 15:00:39.638972 23406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:39.653460 23406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:39.655702 23406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:39.655834 23406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:39.656055 23406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:39.656360 23406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:39.656391 23406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.656422 23406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:39.656445 23406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:39.661996 23406 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:42585
I20260925 15:00:39.662078 23534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:42585 every 8 connection(s)
I20260925 15:00:39.662446 23406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:39.663507 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.663129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:38859)
0925 15:00:39.663283 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.663356 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260925 15:00:39.663926 23535 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:42091 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:42091: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
I20260925 15:00:39.664181 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.664097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:48679)
0925 15:00:39.664137 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.664164 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:00:39.664446 23537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:39.664385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:49037)
0925 15:00:39.664415 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:39.664434 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 15:00:39.664536 23535 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:00:39.664925 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23406
I20260925 15:00:40.506323 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.506093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:54019)
0925 15:00:40.506213 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:40.506260 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260925 15:00:40.534722 23540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.534489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:37615)
0925 15:00:40.534629 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:40.534666 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":53,"threads_started":1}
I20260925 15:00:40.665091 23541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.664855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:51653)
0925 15:00:40.664985 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:40.665030 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260925 15:00:40.769043 23542 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:40.769173 23542 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:40.769455 23542 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
Sep 25 15:00:40 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.194@KRBTEST.COM
Sep 25 15:00:40 dist-test-slave-15hz krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348439, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.195@KRBTEST.COM
I20260925 15:00:40.779466 23539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.769639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:37761 (local address 127.8.13.193:51849)
0925 15:00:40.769713 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:40.769724 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:40.769935 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:40.770638 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:40.770640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:40.771005 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:40.771187 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:40.771193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.772323 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.772324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:40.773157 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:40.773160 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.773725 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:40.773834 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:40.774391 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:40.774400 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:40.776001 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:40.777844 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:40.777852 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:40.777855 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:40.778129 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:40.778334 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778335 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:40.778335 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:40.778462 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:40.778928 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:40.778934 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:40.779052 (+ 118us) client_negotiation.cc:769] Sending connection context
0925 15:00:40.779160 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:00:40.779345 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":49}
I20260925 15:00:40.779582 23544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.769766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:51849 (local address 127.8.13.194:37761)
0925 15:00:40.769920 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:40.769931 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:40.769960 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:00:40.770401 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:40.770403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:40.770444 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:40.770552 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:40.771312 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.772224 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773241 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.773641 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773738 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:40.776149 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:40.776153 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:40.776154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:40.776168 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:40.777761 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778204 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:40.778207 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:40.778207 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:40.778289 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778529 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:40.778530 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:40.778530 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:40.778862 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:40.778942 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:40.779230 (+ 288us) server_negotiation.cc:299] Negotiation successful
0925 15:00:40.779439 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":50,"threads_started":1}
I20260925 15:00:40.779588 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.769766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:58625 (local address 127.8.13.195:42585)
0925 15:00:40.769901 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:40.769912 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:40.769949 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:00:40.770397 (+ 448us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:40.770399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:40.770444 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:40.770614 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:40.771312 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.772208 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773243 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.773641 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773738 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:40.776481 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:40.776486 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:40.776486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:40.776500 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:40.778204 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778528 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:40.778531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:40.778531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:40.778665 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778837 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:40.778838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:40.778838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:40.779087 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:40.779148 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:40.779309 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:00:40.779439 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260925 15:00:40.779616 23543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:40.769684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:42585 (local address 127.8.13.193:58625)
0925 15:00:40.769836 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:40.769846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:40.769905 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:40.770638 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:40.770641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:40.771000 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:40.771187 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:40.771193 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.772305 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.772307 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:40.773159 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:40.773162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:40.773724 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:40.773725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:40.773838 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:40.774421 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:40.774425 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:40.776370 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:40.778266 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778266 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:40.778267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:40.778447 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:40.778729 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:40.778729 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:40.778730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:40.778784 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:40.779148 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:40.779149 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:40.779185 (+ 36us) client_negotiation.cc:769] Sending connection context
0925 15:00:40.779246 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:00:40.779413 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":137,"thread_start_us":95,"threads_started":1}
W20260925 15:00:40.780877 23156 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
W20260925 15:00:40.780966 23157 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:40.781028 23157 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab472316eb694e3888f1ceb444891201; no voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f
I20260925 15:00:40.781178 23542 raft_consensus.cc:2726] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:00:41.507081 23546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:41.506874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:39797)
0925 15:00:41.506996 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:41.507033 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":70,"threads_started":1}
I20260925 15:00:41.535403 23547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:41.535175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:34163)
0925 15:00:41.535316 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:41.535359 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":58,"threads_started":1}
I20260925 15:00:41.665874 23548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:41.665624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:59979)
0925 15:00:41.665780 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:41.665815 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":65,"threads_started":1}
I20260925 15:00:42.507869 23549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:42.507605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:58167)
0925 15:00:42.507750 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:42.507796 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260925 15:00:42.536056 23550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:42.535855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:36045)
0925 15:00:42.535974 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:42.536011 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260925 15:00:42.666518 23551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:42.666297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:45473)
0925 15:00:42.666429 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:42.666473 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":57,"threads_started":1}
I20260925 15:00:42.740396 23552 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:42.740495 23552 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:42.740664 23552 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
W20260925 15:00:42.741056 23156 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
W20260925 15:00:42.741139 23157 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:42.741209 23157 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab472316eb694e3888f1ceb444891201; no voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f
I20260925 15:00:42.741276 23552 raft_consensus.cc:2726] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:00:43.508647 23553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:43.508367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:45997)
0925 15:00:43.508541 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:43.508588 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":103,"threads_started":1}
I20260925 15:00:43.536871 23554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:43.536592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:36263)
0925 15:00:43.536778 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:43.536823 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":79,"threads_started":1}
I20260925 15:00:43.667327 23555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:43.667059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:56303)
0925 15:00:43.667232 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:43.667279 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260925 15:00:44.320256 23556 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:44.320356 23556 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:44.320540 23556 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
W20260925 15:00:44.320897 23157 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
W20260925 15:00:44.320989 23156 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:44.321062 23156 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab472316eb694e3888f1ceb444891201; no voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f
I20260925 15:00:44.321125 23556 raft_consensus.cc:2726] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:00:44.509390 23557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:44.509161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:45753)
0925 15:00:44.509299 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:44.509341 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260925 15:00:44.537577 23558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:44.537341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:50981)
0925 15:00:44.537494 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:44.537532 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":57,"threads_started":1}
I20260925 15:00:44.668071 23559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:44.667834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:43287)
0925 15:00:44.667972 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:44.668021 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260925 15:00:45.510088 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:45.509878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:52135)
0925 15:00:45.509996 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:45.510038 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":60,"threads_started":1}
I20260925 15:00:45.538316 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:45.538088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:34135)
0925 15:00:45.538231 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:45.538270 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260925 15:00:45.668805 23562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:45.668598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:60027)
0925 15:00:45.668716 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:45.668759 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":61,"threads_started":1}
I20260925 15:00:46.510895 23563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:46.510594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:55503)
0925 15:00:46.510766 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:46.510825 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":75,"threads_started":1}
I20260925 15:00:46.539132 23564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:46.538894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:51515)
0925 15:00:46.539035 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:46.539082 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260925 15:00:46.669550 23565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:46.669299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:56229)
0925 15:00:46.669461 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:46.669501 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":67,"threads_started":1}
I20260925 15:00:47.511726 23566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:47.511384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:49995)
0925 15:00:47.511628 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:47.511679 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":111,"threads_started":1}
I20260925 15:00:47.539933 23567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:47.539699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:45233)
0925 15:00:47.539830 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:47.539873 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260925 15:00:47.565505 23568 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:47.565605 23568 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:47.565773 23568 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
W20260925 15:00:47.566105 23157 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
W20260925 15:00:47.566188 23156 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:47.566231 23156 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab472316eb694e3888f1ceb444891201; no voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f
I20260925 15:00:47.566303 23568 raft_consensus.cc:2726] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:00:47.670373 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:47.670105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:54617)
0925 15:00:47.670272 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:47.670320 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":65,"threads_started":1}
I20260925 15:00:48.512442 23570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:48.512221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:34055)
0925 15:00:48.512346 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:48.512392 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260925 15:00:48.540656 23571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:48.540359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:37347)
0925 15:00:48.540525 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:48.540575 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":79,"threads_started":1}
I20260925 15:00:48.671173 23572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:48.670918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:55603)
0925 15:00:48.671071 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:48.671123 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":67,"threads_started":1}
I20260925 15:00:49.431052 23573 raft_consensus.cc:482] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:49.431167 23573 raft_consensus.cc:504] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "658742a535bb49f486a7fc9bc17f2c50" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 42585 } } peers { permanent_uuid: "ab472316eb694e3888f1ceb444891201" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 37261 } } peers { permanent_uuid: "b5478a6a6a6c475fbab9b1ed0ffe9d4f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 37761 } }
I20260925 15:00:49.431336 23573 leader_election.cc:288] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585), b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761)
W20260925 15:00:49.431771 23157 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 658742a535bb49f486a7fc9bc17f2c50 (127.8.13.195:42585): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
W20260925 15:00:49.431856 23156 leader_election.cc:341] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b5478a6a6a6c475fbab9b1ed0ffe9d4f (127.8.13.194:37761): Not found: Tablet not found: 3c4579e2019e4bbe876ca696df47ba85
I20260925 15:00:49.431910 23156 leader_election.cc:302] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ab472316eb694e3888f1ceb444891201; no voters: 658742a535bb49f486a7fc9bc17f2c50, b5478a6a6a6c475fbab9b1ed0ffe9d4f
I20260925 15:00:49.431988 23573 raft_consensus.cc:2726] T 3c4579e2019e4bbe876ca696df47ba85 P ab472316eb694e3888f1ceb444891201 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:00:49.513223 23574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:49.512952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:42725)
0925 15:00:49.513113 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:49.513171 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":60,"threads_started":1}
I20260925 15:00:49.541335 23575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:49.541117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:35953)
0925 15:00:49.541250 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:49.541287 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":78,"threads_started":1}
I20260925 15:00:49.672056 23576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:49.671782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:39945)
0925 15:00:49.671951 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:49.672003 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":63,"threads_started":1}
I20260925 15:00:50.513959 23577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:50.513724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:60195)
0925 15:00:50.513860 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:50.513909 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260925 15:00:50.541993 23578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:50.541778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:43169)
0925 15:00:50.541902 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:50.541945 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":71,"threads_started":1}
I20260925 15:00:50.672905 23579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:50.672663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:60783)
0925 15:00:50.672809 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:50.672855 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":70,"threads_started":1}
I20260925 15:00:51.514776 23580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:51.514532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:34075)
0925 15:00:51.514671 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:51.514717 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":66,"threads_started":1}
I20260925 15:00:51.542703 23581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:51.542484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:36643)
0925 15:00:51.542623 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:51.542657 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":65,"threads_started":1}
I20260925 15:00:51.673774 23582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:51.673498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:54403)
0925 15:00:51.673673 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:51.673722 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":76,"threads_started":1}
I20260925 15:00:52.515604 23583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:52.515256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:43711)
0925 15:00:52.515440 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:52.515484 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260925 15:00:52.543411 23584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:52.543202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:60109)
0925 15:00:52.543318 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:52.543364 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260925 15:00:52.674491 23585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:52.674260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:35101)
0925 15:00:52.674397 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:52.674438 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260925 15:00:53.516333 23586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:53.516122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:46049)
0925 15:00:53.516243 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:53.516287 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":61,"threads_started":1}
I20260925 15:00:53.544088 23587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:53.543864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:46657)
0925 15:00:53.544005 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:53.544042 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260925 15:00:53.675186 23588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:53.674969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:48535)
0925 15:00:53.675095 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:53.675139 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260925 15:00:54.517232 23589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:54.516961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.193:36141)
0925 15:00:54.517138 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:54.517176 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":90,"threads_started":1}
I20260925 15:00:54.544898 23590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:54.544663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.194:46121)
0925 15:00:54.544805 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:54.544849 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260925 15:00:54.672377 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23110
I20260925 15:00:54.675987 23591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:54.675742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42091 (local address 127.8.13.195:57771)
0925 15:00:54.675883 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:54.675938 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:42091: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260925 15:00:54.676847 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23273
I20260925 15:00:54.680739 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23406
2026-09-25T15:00:54Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18082 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:00:54 dist-test-slave-15hz krb5kdc[23594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:00:54 dist-test-slave-15hz krb5kdc[23594](info): set up 2 sockets
Sep 25 15:00:54 dist-test-slave-15hz krb5kdc[23594](info): commencing operation
krb5kdc: starting...
W20260925 15:00:55.631372 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:00:55 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:00:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:00:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:00:55.749084 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:55.830608 23607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:55.830924 23607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:55.830964 23607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:55.832741 23607 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:00:55.832814 23607 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:00:55.832834 23607 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:00:55.832852 23607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:00:55.832892 23607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:55.832921 23607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:55.832944 23607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:00:55.832968 23607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:00:55.834846 23607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:55.835134 23607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:55.835386 23607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:55.840795 23612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:55.840806 23615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:55.840847 23607 server_base.cc:1034] running on GCE node
W20260925 15:00:55.840795 23613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:55.841275 23607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:55.841609 23607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:55.842794 23607 hybrid_clock.cc:648] HybridClock initialized: now 1790348455842776 us; error 34 us; skew 500 ppm
Sep 25 15:00:55 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348455, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:55.849656 23607 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:00:55.850126 23607 webserver.cc:460] Webserver started at http://127.8.13.254:44791/ using document root <none> and password file <none>
I20260925 15:00:55.850340 23607 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:55.850383 23607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:55.850474 23607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:55.851631 23607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "788f5496c2f94536a97e5ed901c0fe98"
format_stamp: "Formatted at 2026-09-25 15:00:55 on dist-test-slave-15hz"
server_key: "66a58a4a4889620b5a124193b48cab3d"
server_key_iv: "b14daf5d228fb19cc1a18885f23c8711"
server_key_version: "encryptionkey@0"
I20260925 15:00:55.851958 23607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "788f5496c2f94536a97e5ed901c0fe98"
format_stamp: "Formatted at 2026-09-25 15:00:55 on dist-test-slave-15hz"
server_key: "66a58a4a4889620b5a124193b48cab3d"
server_key_iv: "b14daf5d228fb19cc1a18885f23c8711"
server_key_version: "encryptionkey@0"
I20260925 15:00:55.853286 23607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:55.854223 23622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:55.854420 23607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:00:55.854519 23607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "788f5496c2f94536a97e5ed901c0fe98"
format_stamp: "Formatted at 2026-09-25 15:00:55 on dist-test-slave-15hz"
server_key: "66a58a4a4889620b5a124193b48cab3d"
server_key_iv: "b14daf5d228fb19cc1a18885f23c8711"
server_key_version: "encryptionkey@0"
I20260925 15:00:55.854591 23607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:55.890218 23607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:55.892602 23607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:55.892728 23607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:55.896492 23607 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:37275
I20260925 15:00:55.896530 23674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:37275 every 8 connection(s)
I20260925 15:00:55.896876 23607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:00:55.897799 23675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:55.900292 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap starting.
I20260925 15:00:55.900916 23675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:55.901191 23675 log.cc:824] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:55.901873 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: No bootstrap required, opened a new log
I20260925 15:00:55.903514 23675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:00:55.903731 23675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:55.903767 23675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Initialized, Role: FOLLOWER
I20260925 15:00:55.903918 23675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:00:55.903983 23675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:00:55.904014 23675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:00:55.904063 23675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:55.904673 23675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:00:55.904748 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23607
I20260925 15:00:55.904778 23675 leader_election.cc:302] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788f5496c2f94536a97e5ed901c0fe98; no voters:
I20260925 15:00:55.904857 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:00:55.904969 23675 leader_election.cc:288] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:00:55.904997 23678 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:55.905061 8247 external_mini_cluster.cc:1428] Setting key 4c8fa06062a3482170386bb99ea68117
I20260925 15:00:55.905114 23678 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 LEADER]: Becoming Leader. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Running, Role: LEADER
I20260925 15:00:55.905246 23675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:00:55.905189 23678 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:00:55.905771 23678 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:00:55.905846 23678 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:00:55.905884 23679 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788f5496c2f94536a97e5ed901c0fe98. Latest consensus state: current_term: 1 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:00:55.905998 23679 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
W20260925 15:00:55.907126 23695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:00:55.907227 23695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:00:55.907289 23688 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:00:55.907424 23688 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:00:55.909775 23688 catalog_manager.cc:1347] Generated new cluster ID: bbf13752eb164f0fab568df01b65b3f0
I20260925 15:00:55.909857 23688 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:00:55 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348455, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:55.917620 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:55.905792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44966 (local address 127.8.13.254:37275)
0925 15:00:55.906115 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:55.906125 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:55.906375 (+ 250us) server_negotiation.cc:407] Connection header received
0925 15:00:55.906522 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:55.906530 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:55.906607 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:55.906751 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:55.907658 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:55.908889 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:55.909718 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:55.910310 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:55.910418 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:55.913377 (+ 2959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:55.913383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:55.913383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:55.913397 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:55.915815 (+ 2418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:55.916258 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:55.916262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:55.916262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:55.916403 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:55.916601 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:55.916602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:55.916602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:55.916955 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:55.917124 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:55.917207 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:00:55.917359 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":81,"threads_started":1}
I20260925 15:00:55.929621 23688 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:00:55.930404 23688 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:00:55.935871 23688 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Generated new TSK 0
I20260925 15:00:55.936124 23688 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:00:55.964082 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:56.047199 23701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:56.047420 23701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:56.047446 23701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:56.048745 23701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:56.048800 23701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:56.048830 23701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:00:56.050408 23701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:56.050681 23701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:56.050909 23701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:56.056123 23707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:56.056159 23706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.056283 23701 server_base.cc:1034] running on GCE node
W20260925 15:00:56.056128 23709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.056622 23701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:56.056888 23701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:56.058034 23701 hybrid_clock.cc:648] HybridClock initialized: now 1790348456058020 us; error 30 us; skew 500 ppm
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:56.060626 23701 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:00:56.061066 23701 webserver.cc:460] Webserver started at http://127.8.13.193:41277/ using document root <none> and password file <none>
I20260925 15:00:56.061291 23701 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:56.061339 23701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:56.061465 23701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:56.062542 23701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "aa0ef5fe2fef43e9919487bea2944046"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "e4d2d40e5dc9d33965088506a4449836"
server_key_iv: "f25c73d0f3d26292f45d00e255a4706d"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.062878 23701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "aa0ef5fe2fef43e9919487bea2944046"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "e4d2d40e5dc9d33965088506a4449836"
server_key_iv: "f25c73d0f3d26292f45d00e255a4706d"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.064239 23701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:00:56.064955 23716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.065155 23701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:00:56.065249 23701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "aa0ef5fe2fef43e9919487bea2944046"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "e4d2d40e5dc9d33965088506a4449836"
server_key_iv: "f25c73d0f3d26292f45d00e255a4706d"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.065316 23701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:56.096690 23701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:56.099138 23701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:56.099292 23701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:56.099555 23701 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:56.099910 23701 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:56.099944 23701 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.099973 23701 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:56.099989 23701 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.105831 23701 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:36597
I20260925 15:00:56.105877 23829 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:36597 every 8 connection(s)
I20260925 15:00:56.106240 23701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:00:56.109347 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23701
I20260925 15:00:56.109459 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:00:56.109649 8247 external_mini_cluster.cc:1428] Setting key cef8fe2477e3f9134f22af2c8e6eb21c
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:56.117341 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.106990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:44097 (local address 127.8.13.254:37275)
0925 15:00:56.107047 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.107049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.107270 (+ 221us) server_negotiation.cc:407] Connection header received
0925 15:00:56.107841 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.107842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.107855 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.107906 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:56.108890 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.109569 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.110932 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.111215 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.111292 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.114010 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:56.114012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:56.114012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:56.114023 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:56.115776 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.116224 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.116226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.116226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.116301 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.116644 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.116645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.116646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.116799 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:56.116867 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.117131 (+ 264us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.117233 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260925 15:00:56.117473 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.107007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:44097)
0925 15:00:56.107152 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.107189 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.107740 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.107980 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.107985 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.108571 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:56.108783 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.108790 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.109669 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.109670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.110827 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.110831 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.111348 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.111349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.111563 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.112161 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:56.112173 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:56.113884 (+ 1711us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:56.115867 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.115879 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.115882 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.116142 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.116434 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.116435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.116435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.116571 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.116864 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:56.116867 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:56.117005 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.117091 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.117241 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 15:00:56.118043 23830 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:00:56.118144 23830 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:56.118327 23830 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:00:56.118881 23639 ts_manager.cc:194] Registered new tserver with Master: aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597)
I20260925 15:00:56.119522 23639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:44097
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:00:56.135445 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:56.216662 23835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:56.216905 23835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:56.216939 23835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:56.218222 23835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:56.218281 23835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:56.218319 23835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:00:56.219929 23835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:56.220201 23835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:56.220430 23835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:56.225628 23843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:56.225613 23841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.225643 23835 server_base.cc:1034] running on GCE node
W20260925 15:00:56.225613 23840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.226065 23835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:56.226310 23835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:56.227497 23835 hybrid_clock.cc:648] HybridClock initialized: now 1790348456227481 us; error 31 us; skew 500 ppm
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:56.230134 23835 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:00:56.230618 23835 webserver.cc:460] Webserver started at http://127.8.13.194:34653/ using document root <none> and password file <none>
I20260925 15:00:56.230825 23835 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:56.230885 23835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:56.231019 23835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:56.232373 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "bc9c60414532419d9c102e0e41ecca25"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "66f982fd23cf4f93e84af27ae214d498"
server_key_iv: "3f1ba13a03295d9e053896f6c52cf80b"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.232765 23835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "bc9c60414532419d9c102e0e41ecca25"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "66f982fd23cf4f93e84af27ae214d498"
server_key_iv: "3f1ba13a03295d9e053896f6c52cf80b"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.234071 23835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:00:56.234855 23850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.235020 23835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:56.235102 23835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "bc9c60414532419d9c102e0e41ecca25"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "66f982fd23cf4f93e84af27ae214d498"
server_key_iv: "3f1ba13a03295d9e053896f6c52cf80b"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.235172 23835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:56.261535 23835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:56.263785 23835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:56.263940 23835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:56.264194 23835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:56.264521 23835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:56.264562 23835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.264592 23835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:56.264609 23835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.270258 23835 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:38507
I20260925 15:00:56.270320 23963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:38507 every 8 connection(s)
I20260925 15:00:56.270645 23835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:00:56.270919 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23835
I20260925 15:00:56.271027 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:00:56.271250 8247 external_mini_cluster.cc:1428] Setting key 4cd3a8d709e565b9c260d850c83efeb2
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 15:00:56.280565 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.271408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:57447 (local address 127.8.13.254:37275)
0925 15:00:56.271661 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.271664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.271735 (+ 71us) server_negotiation.cc:407] Connection header received
0925 15:00:56.272262 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.272263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.272274 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.272325 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:56.273294 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.273821 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.274833 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.275100 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.275160 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.277440 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:56.277442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:56.277442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:56.277451 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:56.278929 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.279564 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.279565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.279566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.279625 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.279867 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.279868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.279868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.279994 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:56.280049 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.280362 (+ 313us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.280465 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260925 15:00:56.280758 23966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.271389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:57447)
0925 15:00:56.271617 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.271666 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.272170 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.272397 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.272400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.272967 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:56.273203 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.273208 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.273911 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.273912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.274740 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.274743 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.275180 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.275181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.275305 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.275912 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:56.275921 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:56.277331 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:56.279019 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.279029 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.279032 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.279287 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.279689 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.279690 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.279690 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.279810 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.280051 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:56.280059 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:56.280233 (+ 174us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.280341 (+ 108us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.280530 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
I20260925 15:00:56.281311 23964 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:00:56.281423 23964 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:56.281620 23964 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:00:56.282033 23639 ts_manager.cc:194] Registered new tserver with Master: bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507)
I20260925 15:00:56.282511 23639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:57447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:00:56.296425 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:00:56.376569 23969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:00:56.376773 23969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:00:56.376796 23969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:00:56.377902 23969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:00:56.377942 23969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:00:56.377966 23969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:00:56.379338 23969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:00:56.379570 23969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:00:56.379773 23969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:00:56.384493 23975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:00:56.384567 23977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.384620 23969 server_base.cc:1034] running on GCE node
W20260925 15:00:56.384469 23974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:00:56.384889 23969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:00:56.385098 23969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:00:56.386243 23969 hybrid_clock.cc:648] HybridClock initialized: now 1790348456386231 us; error 27 us; skew 500 ppm
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:00:56.388765 23969 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:00:56.389218 23969 webserver.cc:460] Webserver started at http://127.8.13.195:40631/ using document root <none> and password file <none>
I20260925 15:00:56.389429 23969 fs_manager.cc:359] Metadata directory not provided
I20260925 15:00:56.389503 23969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:00:56.389629 23969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:00:56.390666 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "c11cefa89d6748769dba4f476cf0cba1"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "0e1376c01387f1cb9995c5b05a56cd8b"
server_key_iv: "4da9231ba7c265a7c76997bc6ceebcd9"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.390982 23969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "c11cefa89d6748769dba4f476cf0cba1"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "0e1376c01387f1cb9995c5b05a56cd8b"
server_key_iv: "4da9231ba7c265a7c76997bc6ceebcd9"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.392303 23969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:00:56.393050 23984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.393240 23969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:00:56.393326 23969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "c11cefa89d6748769dba4f476cf0cba1"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "0e1376c01387f1cb9995c5b05a56cd8b"
server_key_iv: "4da9231ba7c265a7c76997bc6ceebcd9"
server_key_version: "encryptionkey@0"
I20260925 15:00:56.393384 23969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:00:56.407691 23969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:00:56.409672 23969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:00:56.409787 23969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:00:56.409971 23969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:00:56.410233 23969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:00:56.410260 23969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.410281 23969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:00:56.410291 23969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:00:56.415863 23969 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:44239
I20260925 15:00:56.415908 24097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:44239 every 8 connection(s)
I20260925 15:00:56.416236 23969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:00:56.421336 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 23969
I20260925 15:00:56.421489 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:00:56.421660 8247 external_mini_cluster.cc:1428] Setting key 24395cea39addbe1b3bfef9a707ce7a1
Sep 25 15:00:56 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348456, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:00:56.426038 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.417024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:39125 (local address 127.8.13.254:37275)
0925 15:00:56.417103 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.417105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.417246 (+ 141us) server_negotiation.cc:407] Connection header received
0925 15:00:56.417783 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.417785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.417796 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.417847 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:56.418840 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.419375 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.420552 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.420748 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.420808 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.423285 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:56.423287 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:56.423287 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:56.423295 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:56.424608 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.425023 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.425023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.425024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.425076 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.425317 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.425318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.425318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.425475 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:56.425537 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.425883 (+ 346us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.425966 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260925 15:00:56.426175 24100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.416965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:39125)
0925 15:00:56.417125 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.417165 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.417694 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.417913 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.417916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.418499 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:00:56.418717 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.418723 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.419500 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.419501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.420452 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.420456 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.420843 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.420844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.420977 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.421623 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:00:56.421632 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:00:56.423197 (+ 1565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:00:56.424698 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.424704 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.424706 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.424953 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.425129 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:00:56.425129 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:00:56.425129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:00:56.425232 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:00:56.425568 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:00:56.425574 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:00:56.425712 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.425812 (+ 100us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.425956 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260925 15:00:56.426765 24098 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:00:56.426868 24098 heartbeater.cc:464] Registering TS with master...
I20260925 15:00:56.427071 24098 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:00:56.427603 23639 ts_manager.cc:194] Registered new tserver with Master: c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
I20260925 15:00:56.427971 23639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:39125
I20260925 15:00:56.433777 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:00:56.441545 23687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.435253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44968 (local address 127.8.13.254:37275)
0925 15:00:56.435358 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.435360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.435424 (+ 64us) server_negotiation.cc:407] Connection header received
0925 15:00:56.435521 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.435522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.435558 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.435613 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:00:56.436255 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.436778 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.437678 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.437885 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.437969 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.439193 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:00:56.439194 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:00:56.439195 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:00:56.439202 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:00:56.440481 (+ 1279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.440841 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.440842 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.440843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.440893 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:00:56.441048 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:00:56.441049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:00:56.441049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:00:56.441189 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:00:56.441265 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.441412 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.441475 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:00:56.443238 23639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44968:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:00:56.444911 23639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:00:56.451392 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49064 (local address 127.8.13.194:38507)
0925 15:00:56.447607 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.447613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.447622 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:56.447660 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.447661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.447676 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.447740 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:56.447896 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.448983 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.450243 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.450973 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451045 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451107 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.451264 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.451308 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":89,"threads_started":1}
I20260925 15:00:56.451385 24110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:38507 (local address 127.0.0.1:49064)
0925 15:00:56.447383 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.447424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.447485 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.447749 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.447753 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.447755 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:56.447829 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.447831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.449104 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.449105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.450159 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.450162 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.451055 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.451146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451161 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.451238 (+ 77us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.451292 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":36,"thread_start_us":55,"threads_started":1}
I20260925 15:00:56.451747 24109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:44239 (local address 127.0.0.1:45544)
0925 15:00:56.447811 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.447820 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.447875 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.448125 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.448126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.448128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:56.448180 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.448182 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.449087 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.449088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.450301 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.450304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.451332 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451333 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.451400 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451404 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.451634 (+ 230us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.451681 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"mutex_wait_us":39,"thread_start_us":82,"threads_started":1}
I20260925 15:00:56.451917 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:36597 (local address 127.0.0.1:55428)
0925 15:00:56.447958 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.447965 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.448075 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.448324 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.448325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.448326 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:56.448356 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.448357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.449567 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.449568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.450556 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.450558 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.451470 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.451759 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451764 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.451815 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.451854 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":725,"thread_start_us":103,"threads_started":1}
I20260925 15:00:56.451982 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45544 (local address 127.8.13.195:44239)
0925 15:00:56.447770 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.447774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.447878 (+ 104us) server_negotiation.cc:407] Connection header received
0925 15:00:56.447957 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.447959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.447978 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.448045 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:56.448253 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.448993 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.450389 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.451189 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451246 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451298 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.451860 (+ 562us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.451907 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":109,"threads_started":1}
I20260925 15:00:56.452175 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.447608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55428 (local address 127.8.13.193:36597)
0925 15:00:56.447744 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.447749 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.448026 (+ 277us) server_negotiation.cc:407] Connection header received
0925 15:00:56.448168 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.448171 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.448186 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.448248 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:56.448426 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.449463 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.450625 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.451318 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.451374 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.451425 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.452055 (+ 630us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.452104 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260925 15:00:56.452219 23898 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa33a238bc422fa27a9129c1184e6b (DEFAULT_TABLE table=test-table [id=bfe3fcbf37864f7694a817f0b26028f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:56.452575 23898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aa33a238bc422fa27a9129c1184e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:56.452610 24032 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa33a238bc422fa27a9129c1184e6b (DEFAULT_TABLE table=test-table [id=bfe3fcbf37864f7694a817f0b26028f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:56.452736 23764 tablet_service.cc:1467] Processing CreateTablet for tablet e0aa33a238bc422fa27a9129c1184e6b (DEFAULT_TABLE table=test-table [id=bfe3fcbf37864f7694a817f0b26028f7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:00:56.452924 24032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aa33a238bc422fa27a9129c1184e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:56.453011 23764 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0aa33a238bc422fa27a9129c1184e6b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:00:56.455232 24115 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Bootstrap starting.
I20260925 15:00:56.456049 24115 tablet_bootstrap.cc:654] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:56.456156 24116 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Bootstrap starting.
I20260925 15:00:56.456612 24115 log.cc:824] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:56.456779 24116 tablet_bootstrap.cc:654] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:56.456998 24117 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Bootstrap starting.
I20260925 15:00:56.457404 24115 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: No bootstrap required, opened a new log
I20260925 15:00:56.457396 24116 log.cc:824] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:56.457517 24115 ts_tablet_manager.cc:1397] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:00:56.457664 24117 tablet_bootstrap.cc:654] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Neither blocks nor log segments found. Creating new log.
I20260925 15:00:56.458074 24117 log.cc:824] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:00:56.458765 24117 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: No bootstrap required, opened a new log
I20260925 15:00:56.458832 24117 ts_tablet_manager.cc:1397] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:56.458835 24116 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: No bootstrap required, opened a new log
I20260925 15:00:56.458897 24116 ts_tablet_manager.cc:1397] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:00:56.459470 24115 raft_consensus.cc:348] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.459741 24115 raft_consensus.cc:374] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:56.459784 24115 raft_consensus.cc:729] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9c60414532419d9c102e0e41ecca25, State: Initialized, Role: FOLLOWER
I20260925 15:00:56.459903 24115 consensus_queue.cc:260] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.460183 24115 ts_tablet_manager.cc:1428] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:00:56.460544 24116 raft_consensus.cc:348] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.460680 24116 raft_consensus.cc:374] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:56.460712 24116 raft_consensus.cc:729] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0ef5fe2fef43e9919487bea2944046, State: Initialized, Role: FOLLOWER
I20260925 15:00:56.460666 24117 raft_consensus.cc:348] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.460794 24117 raft_consensus.cc:374] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:00:56.460829 24117 raft_consensus.cc:729] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11cefa89d6748769dba4f476cf0cba1, State: Initialized, Role: FOLLOWER
I20260925 15:00:56.460820 24116 consensus_queue.cc:260] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.460937 24117 consensus_queue.cc:260] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.461087 24116 ts_tablet_manager.cc:1428] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260925 15:00:56.461104 23830 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
I20260925 15:00:56.461211 24117 ts_tablet_manager.cc:1428] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:00:56.461217 24098 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
I20260925 15:00:56.461414 23964 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
I20260925 15:00:56.501083 24122 raft_consensus.cc:482] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:00:56.501209 24122 raft_consensus.cc:504] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.501601 24122 leader_election.cc:288] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
I20260925 15:00:56.504741 24113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.501867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43329 (local address 127.8.13.195:44239)
0925 15:00:56.501916 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.501918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.501949 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:00:56.502027 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.502028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.502038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.502091 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:56.502329 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.502863 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.503901 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.504359 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.504417 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.504424 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.504628 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.504673 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:00:56.504750 24112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.501919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:56239 (local address 127.8.13.194:38507)
0925 15:00:56.501962 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.501963 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.502142 (+ 179us) server_negotiation.cc:407] Connection header received
0925 15:00:56.502185 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.502186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.502199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.502259 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:00:56.502494 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.502980 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.503995 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.504410 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.504488 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.504497 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.504645 (+ 148us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.504689 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:00:56.504750 24124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.501803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:38507 (local address 127.8.13.193:56239)
0925 15:00:56.502070 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.502080 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.502134 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.502307 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.502308 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.502311 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:56.502387 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.502388 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.503072 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.503073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.503892 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.503895 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.504493 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.504494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.504547 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.504550 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.504596 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.504640 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":95,"thread_start_us":111,"threads_started":1}
I20260925 15:00:56.504750 23832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.501803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:44239 (local address 127.8.13.193:43329)
0925 15:00:56.501875 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.501888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:00:56.501951 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:00:56.502111 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:00:56.502112 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:00:56.502114 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:00:56.502195 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.502197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.502951 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.502952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.503800 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:00:56.503803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.504454 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.504455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:00:56.504545 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.504549 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:00:56.504610 (+ 61us) client_negotiation.cc:240] Negotiation successful
0925 15:00:56.504659 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260925 15:00:56.505141 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aa33a238bc422fa27a9129c1184e6b" candidate_uuid: "aa0ef5fe2fef43e9919487bea2944046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11cefa89d6748769dba4f476cf0cba1" is_pre_election: true
I20260925 15:00:56.505124 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aa33a238bc422fa27a9129c1184e6b" candidate_uuid: "aa0ef5fe2fef43e9919487bea2944046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9c60414532419d9c102e0e41ecca25" is_pre_election: true
I20260925 15:00:56.505287 24052 raft_consensus.cc:2454] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa0ef5fe2fef43e9919487bea2944046 in term 0.
I20260925 15:00:56.505287 23918 raft_consensus.cc:2454] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aa0ef5fe2fef43e9919487bea2944046 in term 0.
I20260925 15:00:56.505561 23720 leader_election.cc:302] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046, bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1; no voters:
I20260925 15:00:56.505720 24122 raft_consensus.cc:2781] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:00:56.505781 24122 raft_consensus.cc:482] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:00:56.505803 24122 raft_consensus.cc:3037] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:56.506516 24122 raft_consensus.cc:504] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.506680 24122 leader_election.cc:288] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 1 election: Requested vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
I20260925 15:00:56.506860 23918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aa33a238bc422fa27a9129c1184e6b" candidate_uuid: "aa0ef5fe2fef43e9919487bea2944046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc9c60414532419d9c102e0e41ecca25"
I20260925 15:00:56.506865 24052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0aa33a238bc422fa27a9129c1184e6b" candidate_uuid: "aa0ef5fe2fef43e9919487bea2944046" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11cefa89d6748769dba4f476cf0cba1"
I20260925 15:00:56.506960 24052 raft_consensus.cc:3037] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:56.506960 23918 raft_consensus.cc:3037] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:00:56.507936 23918 raft_consensus.cc:2454] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0ef5fe2fef43e9919487bea2944046 in term 1.
I20260925 15:00:56.507936 24052 raft_consensus.cc:2454] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aa0ef5fe2fef43e9919487bea2944046 in term 1.
I20260925 15:00:56.508167 23720 leader_election.cc:302] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046, bc9c60414532419d9c102e0e41ecca25; no voters:
I20260925 15:00:56.508271 24122 raft_consensus.cc:2781] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:00:56.508456 24122 raft_consensus.cc:686] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 LEADER]: Becoming Leader. State: Replica: aa0ef5fe2fef43e9919487bea2944046, State: Running, Role: LEADER
I20260925 15:00:56.508669 24122 consensus_queue.cc:237] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:00:56.509565 23639 catalog_manager.cc:5495] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "aa0ef5fe2fef43e9919487bea2944046" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } health_report { overall_health: HEALTHY } } }
W20260925 15:00:56.521230 23965 tablet.cc:2369] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:00:56.528296 24114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.525929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55432 (local address 127.8.13.193:36597)
0925 15:00:56.525980 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:00:56.525983 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:00:56.525992 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:00:56.526069 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:00:56.526069 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:00:56.526090 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:00:56.526149 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:00:56.526269 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.526749 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.527519 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:00:56.527740 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:00:56.527797 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:00:56.527952 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:00:56.528065 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:00:56.528139 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:00:56.528166 (+ 27us) server_negotiation.cc:299] Negotiation successful
0925 15:00:56.528189 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:00:56.531067 23918 raft_consensus.cc:1261] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Refusing update from remote peer aa0ef5fe2fef43e9919487bea2944046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:56.531068 24052 raft_consensus.cc:1261] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Refusing update from remote peer aa0ef5fe2fef43e9919487bea2944046: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:00:56.531395 24122 consensus_queue.cc:1035] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:56.531606 24125 consensus_queue.cc:1035] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:00:56.534557 24129 mvcc.cc:205] Tried to move back new op lower bound from 7333267277948559360 to 7333267277860306944. Current Snapshot: MvccSnapshot[applied={T|T < 7333267277948559360 or (T in {7333267277948559360})}]
I20260925 15:00:56.536865 23637 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:56.536962 23637 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44968:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:00:56.537781 23637 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Sending DeleteTablet for 3 replicas of tablet e0aa33a238bc422fa27a9129c1184e6b
I20260925 15:00:56.538187 24032 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aa33a238bc422fa27a9129c1184e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:56 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:45544
I20260925 15:00:56.538249 23764 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aa33a238bc422fa27a9129c1184e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:56 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:55428
I20260925 15:00:56.538463 23898 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aa33a238bc422fa27a9129c1184e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:56 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:49064
I20260925 15:00:56.538478 24137 tablet_replica.cc:330] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: stopping tablet replica
I20260925 15:00:56.538614 24137 raft_consensus.cc:2229] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:56.538718 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23701
I20260925 15:00:56.538734 24137 raft_consensus.cc:2258] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:56.539649 24137 ts_tablet_manager.cc:1905] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:56.540447 24139 tablet_replica.cc:330] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: stopping tablet replica
I20260925 15:00:56.540570 24139 raft_consensus.cc:2229] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:00:56.540685 24139 raft_consensus.cc:2258] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:00:56.541133 24139 ts_tablet_manager.cc:1905] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:00:56.541400 24137 ts_tablet_manager.cc:1918] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:56.541474 24137 log.cc:1196] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/e0aa33a238bc422fa27a9129c1184e6b
I20260925 15:00:56.541746 24137 ts_tablet_manager.cc:1939] T e0aa33a238bc422fa27a9129c1184e6b P c11cefa89d6748769dba4f476cf0cba1: Deleting consensus metadata
I20260925 15:00:56.542246 23623 catalog_manager.cc:4841] TS c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): tablet e0aa33a238bc422fa27a9129c1184e6b (table test-table [id=bfe3fcbf37864f7694a817f0b26028f7]) successfully deleted
I20260925 15:00:56.542668 24139 ts_tablet_manager.cc:1918] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:00:56.542727 24139 log.cc:1196] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/e0aa33a238bc422fa27a9129c1184e6b
I20260925 15:00:56.542943 24139 ts_tablet_manager.cc:1939] T e0aa33a238bc422fa27a9129c1184e6b P bc9c60414532419d9c102e0e41ecca25: Deleting consensus metadata
I20260925 15:00:56.543428 23626 catalog_manager.cc:4841] TS bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): tablet e0aa33a238bc422fa27a9129c1184e6b (table test-table [id=bfe3fcbf37864f7694a817f0b26028f7]) successfully deleted
W20260925 15:00:56.543715 23624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:36597 (error 108)
I20260925 15:00:56.543856 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23835
I20260925 15:00:56.545646 24111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:00:56.545492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:36597 (local address 127.0.0.1:55444)
0925 15:00:56.545559 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:00:56.545611 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260925 15:00:56.545778 23624 catalog_manager.cc:4584] TS aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0aa33a238bc422fa27a9129c1184e6b: Network error: Client connection negotiation failed: client connection to 127.8.13.193:36597: connect: Connection refused (error 111)
I20260925 15:00:56.548353 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23969
I20260925 15:00:56.552758 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 23607
I20260925 15:01:16.557724 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--webserver_interface=127.8.13.254
--webserver_port=44791
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:16.640342 24141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:16.640575 24141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:16.640599 24141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:16.641824 24141 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:01:16.641873 24141 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:01:16.641885 24141 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:01:16.641896 24141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:16.641908 24141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:16.641920 24141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:16.641932 24141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:16.641943 24141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:16.643597 24141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:16.643867 24141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:16.644111 24141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:16.652658 24147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.652654 24148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.652655 24150 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:16.652905 24141 server_base.cc:1034] running on GCE node
I20260925 15:01:16.653190 24141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:16.653509 24141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:16.654668 24141 hybrid_clock.cc:648] HybridClock initialized: now 1790348476654673 us; error 60 us; skew 500 ppm
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:16.657313 24141 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:01:16.657797 24141 webserver.cc:460] Webserver started at http://127.8.13.254:44791/ using document root <none> and password file <none>
I20260925 15:01:16.658021 24141 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:16.658085 24141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:16.659508 24141 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:16.660185 24157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.660343 24141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:16.660426 24141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "788f5496c2f94536a97e5ed901c0fe98"
format_stamp: "Formatted at 2026-09-25 15:00:55 on dist-test-slave-15hz"
server_key: "66a58a4a4889620b5a124193b48cab3d"
server_key_iv: "b14daf5d228fb19cc1a18885f23c8711"
server_key_version: "encryptionkey@0"
I20260925 15:01:16.660822 24141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:16.675974 24141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:16.677947 24141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:16.678089 24141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:16.681919 24141 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:37275
I20260925 15:01:16.681965 24209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:37275 every 8 connection(s)
I20260925 15:01:16.682351 24141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:16.682740 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24141
I20260925 15:01:16.683058 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:36597
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=41277
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:16.684240 24210 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:01:16.684954 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap starting.
I20260925 15:01:16.693092 24210 log.cc:824] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:16.695104 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:16.695402 24210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap complete.
I20260925 15:01:16.697558 24210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:16.697796 24210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Initialized, Role: FOLLOWER
I20260925 15:01:16.697940 24210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:16.697999 24210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:16.698045 24210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:16.698091 24210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 1 FOLLOWER]: Advancing to term 2
I20260925 15:01:16.699365 24210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:16.699467 24210 leader_election.cc:302] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788f5496c2f94536a97e5ed901c0fe98; no voters:
I20260925 15:01:16.699689 24210 leader_election.cc:288] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260925 15:01:16.699736 24214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Leader election won for term 2
I20260925 15:01:16.699913 24214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 LEADER]: Becoming Leader. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Running, Role: LEADER
I20260925 15:01:16.699973 24210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:16.700032 24214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:16.700493 24214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788f5496c2f94536a97e5ed901c0fe98. Latest consensus state: current_term: 2 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:01:16.700548 24214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:16.700388 24215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:01:16.700700 24215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:16.700796 24223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:16.701609 24223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:16.701754 24223 catalog_manager.cc:1259] Loaded cluster ID: bbf13752eb164f0fab568df01b65b3f0
I20260925 15:01:16.702024 24223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:16.703223 24223 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:16.703481 24223 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Loaded TSK: 0
I20260925 15:01:16.721946 24223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Generated new TSK 1
I20260925 15:01:16.722064 24223 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:01:16.765916 24212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:16.766136 24212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:16.766160 24212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:16.767357 24212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:16.767405 24212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:16.767431 24212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:16.768950 24212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=41277
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:16.769169 24212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:16.769397 24212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:16.774314 24237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.774317 24240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.774314 24238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:16.774765 24212 server_base.cc:1034] running on GCE node
I20260925 15:01:16.774995 24212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:16.775310 24212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:16.776496 24212 hybrid_clock.cc:648] HybridClock initialized: now 1790348476776477 us; error 34 us; skew 500 ppm
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:16.779604 24212 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:01:16.780202 24212 webserver.cc:460] Webserver started at http://127.8.13.193:41277/ using document root <none> and password file <none>
I20260925 15:01:16.780471 24212 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:16.780558 24212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:16.782305 24212 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:16.783057 24247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.783222 24212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:16.783303 24212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "aa0ef5fe2fef43e9919487bea2944046"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "e4d2d40e5dc9d33965088506a4449836"
server_key_iv: "f25c73d0f3d26292f45d00e255a4706d"
server_key_version: "encryptionkey@0"
I20260925 15:01:16.783705 24212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:16.803076 24212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:16.805532 24212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:16.805661 24212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:16.805929 24212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:16.806392 24254 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:01:16.808267 24212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:01:16.808327 24212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:01:16.808365 24212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:01:16.809011 24212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:01:16.809048 24212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:01:16.809136 24254 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Bootstrap starting.
I20260925 15:01:16.816946 24212 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:36597
I20260925 15:01:16.816996 24361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:36597 every 8 connection(s)
I20260925 15:01:16.817474 24212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:16.818331 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24212
I20260925 15:01:16.818624 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:38507
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=34653
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:16.820343 24254 log.cc:824] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:16.822994 24254 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:16.823343 24254 tablet_bootstrap.cc:492] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Bootstrap complete.
I20260925 15:01:16.824451 24254 ts_tablet_manager.cc:1397] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20260925 15:01:16.825971 24254 raft_consensus.cc:348] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:16.826352 24254 raft_consensus.cc:729] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0ef5fe2fef43e9919487bea2944046, State: Initialized, Role: FOLLOWER
I20260925 15:01:16.826529 24254 consensus_queue.cc:260] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:16.826851 24254 ts_tablet_manager.cc:1428] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:16.835711 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.818729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:50727 (local address 127.8.13.254:37275)
0925 15:01:16.818914 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:16.818924 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:16.822979 (+ 4055us) server_negotiation.cc:407] Connection header received
0925 15:01:16.823834 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:16.823843 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:16.823905 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:16.824027 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:16.824955 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.826362 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.827813 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.828667 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.828804 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.831561 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:16.831566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:16.831567 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:16.831579 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:16.833577 (+ 1998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.834164 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.834168 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.834168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.834319 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.834608 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.834609 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.834609 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.834921 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:16.835006 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:16.835344 (+ 338us) server_negotiation.cc:299] Negotiation successful
0925 15:01:16.835481 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260925 15:01:16.835726 24364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.818325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:50727)
0925 15:01:16.822774 (+ 4449us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:16.822827 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:16.823586 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:16.824211 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:16.824215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:16.824564 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:16.824831 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.824837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.826484 (+ 1647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.826485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.827719 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.827721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.828773 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.828774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.828911 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.829560 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:16.829571 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:16.831392 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:16.833671 (+ 2279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.833680 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.833683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.834012 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.834393 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.834394 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.834395 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.834537 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.834992 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:16.834997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:16.835154 (+ 157us) client_negotiation.cc:769] Sending connection context
0925 15:01:16.835280 (+ 126us) client_negotiation.cc:240] Negotiation successful
0925 15:01:16.835461 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4396,"thread_start_us":111,"threads_started":1}
I20260925 15:01:16.836822 24362 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:01:16.836915 24362 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:16.837081 24362 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:01:16.837822 24174 ts_manager.cc:194] Registered new tserver with Master: aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597)
I20260925 15:01:16.839044 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:50727
I20260925 15:01:16.840065 24362 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.193@KRBTEST.COM
I20260925 15:01:16.848452 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.838723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40784 (local address 127.8.13.193:36597)
0925 15:01:16.839068 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:16.839072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:16.839079 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:01:16.839189 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:16.839192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:16.839206 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:16.839269 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:16.840724 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.841683 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.842633 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.843015 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.843069 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.845599 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:16.845609 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:16.845610 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:16.845618 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:16.847158 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.847580 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.847583 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.847584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.847644 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.847817 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.847818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.847819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.847972 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:16.848043 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:16.848243 (+ 200us) server_negotiation.cc:299] Negotiation successful
0925 15:01:16.848342 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":68,"threads_started":1}
I20260925 15:01:16.848452 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.838481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:36597 (local address 127.0.0.1:40784)
0925 15:01:16.838868 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:16.838902 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:16.838962 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:16.839330 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:16.839336 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:16.840474 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:16.840539 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.840543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.841771 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.841772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.842547 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.842550 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.843125 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.843126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.843196 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.843802 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:16.843807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:16.845508 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:16.847258 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.847266 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.847269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.847477 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.847706 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.847707 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.847707 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.847766 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.848036 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:16.848043 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:16.848098 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:01:16.848204 (+ 106us) client_negotiation.cc:240] Negotiation successful
0925 15:01:16.848342 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":93,"threads_started":1}
I20260925 15:01:16.848964 24281 tablet_service.cc:1514] Processing DeleteTablet for tablet e0aa33a238bc422fa27a9129c1184e6b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:00:56 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:40784
I20260925 15:01:16.849470 24371 tablet_replica.cc:330] stopping tablet replica
I20260925 15:01:16.849573 24371 raft_consensus.cc:2229] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:16.849643 24371 raft_consensus.cc:2258] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:16.849929 24371 ts_tablet_manager.cc:1905] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:16.851887 24371 ts_tablet_manager.cc:1918] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:16.851953 24371 log.cc:1196] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal/wals/e0aa33a238bc422fa27a9129c1184e6b
I20260925 15:01:16.852138 24371 ts_tablet_manager.cc:1939] T e0aa33a238bc422fa27a9129c1184e6b P aa0ef5fe2fef43e9919487bea2944046: Deleting consensus metadata
I20260925 15:01:16.852653 24159 catalog_manager.cc:4841] TS aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597): tablet e0aa33a238bc422fa27a9129c1184e6b (table test-table [id=bfe3fcbf37864f7694a817f0b26028f7]) successfully deleted
W20260925 15:01:16.908406 24366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:16.908645 24366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:16.908674 24366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:16.909891 24366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:16.909941 24366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:16.909971 24366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:16.911496 24366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:38507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=34653
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:16.911788 24366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:16.912017 24366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:16.917150 24379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.917173 24376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:16.917155 24377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:16.917392 24366 server_base.cc:1034] running on GCE node
I20260925 15:01:16.917598 24366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:16.917845 24366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:16.919008 24366 hybrid_clock.cc:648] HybridClock initialized: now 1790348476918992 us; error 26 us; skew 500 ppm
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:16.921561 24366 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:01:16.922043 24366 webserver.cc:460] Webserver started at http://127.8.13.194:34653/ using document root <none> and password file <none>
I20260925 15:01:16.922251 24366 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:16.922320 24366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:16.923707 24366 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:01:16.924304 24386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.924455 24366 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.924540 24366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "bc9c60414532419d9c102e0e41ecca25"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "66f982fd23cf4f93e84af27ae214d498"
server_key_iv: "3f1ba13a03295d9e053896f6c52cf80b"
server_key_version: "encryptionkey@0"
I20260925 15:01:16.924891 24366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:16.942835 24366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:16.945132 24366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:16.945291 24366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:16.945533 24366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:16.945844 24366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:16.945881 24366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.945915 24366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:16.945942 24366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:16.952085 24366 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:38507
I20260925 15:01:16.952150 24499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:38507 every 8 connection(s)
I20260925 15:01:16.952512 24366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:16.954823 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24366
I20260925 15:01:16.955116 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:44239
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=40631
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 25 15:01:16 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348476, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:16.962954 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.953257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:47043 (local address 127.8.13.254:37275)
0925 15:01:16.953341 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:16.953343 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:16.953531 (+ 188us) server_negotiation.cc:407] Connection header received
0925 15:01:16.954058 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:16.954059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:16.954074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:16.954137 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:16.955219 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.955977 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.957058 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.957373 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.957438 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.959934 (+ 2496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:16.959936 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:16.959937 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:16.959945 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:16.961500 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.961911 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.961912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.961912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.961977 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:16.962184 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:16.962185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:16.962185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:16.962345 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:16.962418 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:16.962726 (+ 308us) server_negotiation.cc:299] Negotiation successful
0925 15:01:16.962850 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:01:16.963052 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:16.953226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:47043)
0925 15:01:16.953399 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:16.953444 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:16.953958 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:16.954214 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:16.954218 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:16.954850 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:16.955102 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.955108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.956102 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.956103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.956965 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:16.956967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:16.957484 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:16.957485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:16.957629 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:16.958153 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:16.958164 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:16.959820 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:16.961587 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.961595 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.961597 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.961840 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.962039 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:16.962039 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:16.962039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:16.962133 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:16.962434 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:16.962440 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:16.962591 (+ 151us) client_negotiation.cc:769] Sending connection context
0925 15:01:16.962692 (+ 101us) client_negotiation.cc:240] Negotiation successful
0925 15:01:16.962862 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":75,"threads_started":1}
I20260925 15:01:16.963621 24500 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:01:16.963723 24500 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:16.963917 24500 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:01:16.964339 24174 ts_manager.cc:194] Registered new tserver with Master: bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507)
I20260925 15:01:16.964701 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:47043
W20260925 15:01:17.038969 24503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:17.039197 24503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:17.039222 24503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:17.040544 24503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:17.040596 24503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:17.040627 24503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:17.042506 24503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:44239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=40631
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24503
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:17.042766 24503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:17.043007 24503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:17.048172 24508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.048169 24511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:17.048193 24509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:17.048288 24503 server_base.cc:1034] running on GCE node
I20260925 15:01:17.048593 24503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:17.048839 24503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:17.049976 24503 hybrid_clock.cc:648] HybridClock initialized: now 1790348477049959 us; error 38 us; skew 500 ppm
Sep 25 15:01:17 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:17.052436 24503 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:01:17.052937 24503 webserver.cc:460] Webserver started at http://127.8.13.195:40631/ using document root <none> and password file <none>
I20260925 15:01:17.053138 24503 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:17.053203 24503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:17.054594 24503 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:17.055190 24518 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.055344 24503 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:17.055437 24503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "c11cefa89d6748769dba4f476cf0cba1"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "0e1376c01387f1cb9995c5b05a56cd8b"
server_key_iv: "4da9231ba7c265a7c76997bc6ceebcd9"
server_key_version: "encryptionkey@0"
I20260925 15:01:17.055842 24503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:17.075176 24503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:17.077346 24503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:17.077499 24503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:17.077724 24503 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:17.078023 24503 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:17.078056 24503 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.078085 24503 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:17.078107 24503 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:17.083824 24503 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:44239
I20260925 15:01:17.083869 24631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:44239 every 8 connection(s)
I20260925 15:01:17.084239 24503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:01:17 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348477, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:17.090983 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24503
I20260925 15:01:17.093609 24365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.084942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:59673 (local address 127.8.13.254:37275)
0925 15:01:17.085005 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.085007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.085208 (+ 201us) server_negotiation.cc:407] Connection header received
0925 15:01:17.085670 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.085672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.085683 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.085734 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:17.086718 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.087211 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.088228 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.088441 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.088496 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.090512 (+ 2016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:17.090513 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:17.090513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:17.090521 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:17.092120 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.092523 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.092525 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.092525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.092595 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.092886 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.092887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.092887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.093006 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:17.093071 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.093405 (+ 334us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.093497 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:01:17.093627 24634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.084932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:59673)
0925 15:01:17.085076 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.085113 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.085591 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.085796 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.085799 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.086408 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:17.086619 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.086624 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.087280 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.087281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.088156 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.088159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.088504 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.088504 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.088605 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.089068 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:17.089077 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:17.090412 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:17.092193 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:17.092201 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:17.092203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:17.092442 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:17.092659 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:17.092660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:17.092660 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:17.092804 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:17.093068 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:17.093073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:17.093239 (+ 166us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.093323 (+ 84us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.093451 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":78,"threads_started":1}
I20260925 15:01:17.094040 24632 heartbeater.cc:347] Connected to a master server at 127.8.13.254:37275
I20260925 15:01:17.094138 24632 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:17.094319 24632 heartbeater.cc:511] Master 127.8.13.254:37275 requested a full tablet report, sending...
I20260925 15:01:17.094708 24174 ts_manager.cc:194] Registered new tserver with Master: c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
I20260925 15:01:17.095050 24174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:59673
I20260925 15:01:17.097667 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.091393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52728 (local address 127.8.13.254:37275)
0925 15:01:17.091546 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.091548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.091669 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:01:17.091748 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.091749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.091759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.091812 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:17.092476 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.093094 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.094156 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.094396 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.094453 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.095726 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:17.095727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:17.095727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:17.095734 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:17.096866 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.097141 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.097142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.097143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.097188 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.097299 (+ 111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.097299 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.097300 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.097404 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:17.097520 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.097545 (+ 25us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.097603 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260925 15:01:17.098335 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:17.100868 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.098642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52738 (local address 127.8.13.254:37275)
0925 15:01:17.098694 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.098695 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.098789 (+ 94us) server_negotiation.cc:407] Connection header received
0925 15:01:17.098845 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.098845 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.098874 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.098925 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:17.099004 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.099552 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.100244 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.100498 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.100543 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.100655 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:17.100698 (+ 43us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0925 15:01:17.100821 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52738: token expired
Metrics: {"server-negotiator.queue_time_us":33}
W20260925 15:01:17.100914 24635 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52738: token expired
W20260925 15:01:17.100970 24104 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.13.254:37275)
I20260925 15:01:17.106205 24635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.101198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52754 (local address 127.8.13.254:37275)
0925 15:01:17.101227 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.101229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.101236 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:01:17.101271 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.101272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.101280 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.101320 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:17.101757 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.102182 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.102782 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.102938 (+ 156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.102984 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.104009 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:17.104010 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:17.104011 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:17.104016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:17.105339 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.105663 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.105664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.105664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.105714 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:17.105868 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:17.105869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:17.105869 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:17.105959 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:17.106026 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.106087 (+ 61us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.106142 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260925 15:01:17.107316 24174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:17.108947 24174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:17.110945 24281 tablet_service.cc:1467] Processing CreateTablet for tablet 03425cb402964a02b82bacfd7b107529 (DEFAULT_TABLE table=test-table [id=213576ff9e604c8a96cf16ea4e690053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:17.111145 24281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03425cb402964a02b82bacfd7b107529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:17.112540 24254 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Bootstrap starting.
I20260925 15:01:17.112864 24254 tablet_bootstrap.cc:654] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:17.113404 24254 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: No bootstrap required, opened a new log
I20260925 15:01:17.113471 24254 ts_tablet_manager.cc:1397] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260925 15:01:17.113786 24254 raft_consensus.cc:348] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.113844 24254 raft_consensus.cc:374] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:17.113884 24254 raft_consensus.cc:729] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0ef5fe2fef43e9919487bea2944046, State: Initialized, Role: FOLLOWER
I20260925 15:01:17.113930 24254 consensus_queue.cc:260] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.114024 24254 ts_tablet_manager.cc:1428] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260925 15:01:17.114665 24369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.110675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:38507 (local address 127.0.0.1:33934)
0925 15:01:17.110713 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.110723 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.110754 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.111183 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.111184 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.111185 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:17.111234 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.111235 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.112376 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.112377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.113395 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.113398 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.114416 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.114417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.114486 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.114505 (+ 19us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.114550 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.114594 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:01:17.114934 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.110827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33934 (local address 127.8.13.194:38507)
0925 15:01:17.111033 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.111038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.111048 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:17.111088 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.111090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.111105 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.111179 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:17.111333 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.112306 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.113501 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.114252 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.114323 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.114385 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.114814 (+ 429us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.114860 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":100,"threads_started":1}
I20260925 15:01:17.115499 24434 tablet_service.cc:1467] Processing CreateTablet for tablet 03425cb402964a02b82bacfd7b107529 (DEFAULT_TABLE table=test-table [id=213576ff9e604c8a96cf16ea4e690053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:17.115839 24434 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03425cb402964a02b82bacfd7b107529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:17.116474 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.110724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:44239 (local address 127.0.0.1:35432)
0925 15:01:17.110878 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.110887 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.110925 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.112785 (+ 1860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.112786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.112788 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:17.112824 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.112825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.114057 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.114058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.115102 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.115105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.116209 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.116210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.116312 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.116316 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.116366 (+ 50us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.116408 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
I20260925 15:01:17.116499 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.110959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35432 (local address 127.8.13.195:44239)
0925 15:01:17.112615 (+ 1656us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.112620 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.112630 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:17.112668 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.112671 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.112689 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.112775 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:17.112875 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.113893 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.115590 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.116137 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.116200 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.116255 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.116386 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.116423 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1618,"thread_start_us":93,"threads_started":1}
I20260925 15:01:17.117349 24566 tablet_service.cc:1467] Processing CreateTablet for tablet 03425cb402964a02b82bacfd7b107529 (DEFAULT_TABLE table=test-table [id=213576ff9e604c8a96cf16ea4e690053]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:17.117847 24566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03425cb402964a02b82bacfd7b107529. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:17.119407 24642 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Bootstrap starting.
I20260925 15:01:17.120014 24642 tablet_bootstrap.cc:654] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:17.120473 24642 log.cc:824] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:17.120937 24644 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Bootstrap starting.
I20260925 15:01:17.121136 24642 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: No bootstrap required, opened a new log
I20260925 15:01:17.121193 24642 ts_tablet_manager.cc:1397] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:17.121529 24644 tablet_bootstrap.cc:654] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:17.121973 24644 log.cc:824] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:17.122605 24644 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: No bootstrap required, opened a new log
I20260925 15:01:17.122666 24644 ts_tablet_manager.cc:1397] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:17.122624 24642 raft_consensus.cc:348] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.122800 24642 raft_consensus.cc:374] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:17.122848 24642 raft_consensus.cc:729] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc9c60414532419d9c102e0e41ecca25, State: Initialized, Role: FOLLOWER
I20260925 15:01:17.122972 24642 consensus_queue.cc:260] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.123263 24642 ts_tablet_manager.cc:1428] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:17.123263 24500 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
I20260925 15:01:17.124590 24644 raft_consensus.cc:348] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.124733 24644 raft_consensus.cc:374] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:17.124756 24644 raft_consensus.cc:729] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c11cefa89d6748769dba4f476cf0cba1, State: Initialized, Role: FOLLOWER
I20260925 15:01:17.124845 24644 consensus_queue.cc:260] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.125077 24644 ts_tablet_manager.cc:1428] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:01:17.125115 24632 heartbeater.cc:503] Master 127.8.13.254:37275 was elected leader, sending a full tablet report...
I20260925 15:01:17.199899 24646 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:17.200037 24646 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.200423 24646 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239), aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597)
W20260925 15:01:17.203047 24501 tablet.cc:2369] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:17.203671 24648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.200626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:36597 (local address 127.8.13.194:45569)
0925 15:01:17.200817 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.200836 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.200882 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.201038 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.201039 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.201040 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:17.201093 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.201095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.201989 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.201990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.202764 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.202766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.203384 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.203385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.203461 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.203465 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.203544 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.203587 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":2,"thread_start_us":125,"threads_started":1}
I20260925 15:01:17.203660 24370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.200668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:45569 (local address 127.8.13.193:36597)
0925 15:01:17.200891 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.200892 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.200903 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:01:17.200933 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.200934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.200961 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.201024 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:17.201155 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.201904 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.202853 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.203308 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.203378 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.203472 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.203552 (+ 80us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.203593 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197}
I20260925 15:01:17.203753 24502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.200619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:44239 (local address 127.8.13.194:56535)
0925 15:01:17.200708 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.200720 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:17.200769 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:17.200945 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:17.200946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:17.200947 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:17.201009 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.201011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.201946 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.201947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.202761 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:17.202764 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.203491 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.203493 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:17.203602 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.203605 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:17.203647 (+ 42us) client_negotiation.cc:240] Negotiation successful
0925 15:01:17.203683 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":18}
I20260925 15:01:17.203913 24640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.200727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:56535 (local address 127.8.13.195:44239)
0925 15:01:17.200783 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.200785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.200796 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:01:17.200826 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.200827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.200838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.200895 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:17.201086 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.201862 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.202860 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.203394 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.203452 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.203461 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.203808 (+ 347us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.203850 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:01:17.204085 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03425cb402964a02b82bacfd7b107529" candidate_uuid: "bc9c60414532419d9c102e0e41ecca25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0ef5fe2fef43e9919487bea2944046" is_pre_election: true
I20260925 15:01:17.204188 24586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03425cb402964a02b82bacfd7b107529" candidate_uuid: "bc9c60414532419d9c102e0e41ecca25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11cefa89d6748769dba4f476cf0cba1" is_pre_election: true
I20260925 15:01:17.204249 24316 raft_consensus.cc:2454] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9c60414532419d9c102e0e41ecca25 in term 0.
I20260925 15:01:17.204342 24586 raft_consensus.cc:2454] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc9c60414532419d9c102e0e41ecca25 in term 0.
I20260925 15:01:17.204466 24388 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046, bc9c60414532419d9c102e0e41ecca25; no voters:
I20260925 15:01:17.204607 24646 raft_consensus.cc:2781] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:17.204665 24646 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:17.204694 24646 raft_consensus.cc:3037] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:17.205435 24646 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.205561 24646 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [CANDIDATE]: Term 1 election: Requested vote from peers c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239), aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597)
I20260925 15:01:17.205710 24316 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03425cb402964a02b82bacfd7b107529" candidate_uuid: "bc9c60414532419d9c102e0e41ecca25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa0ef5fe2fef43e9919487bea2944046"
I20260925 15:01:17.205758 24586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03425cb402964a02b82bacfd7b107529" candidate_uuid: "bc9c60414532419d9c102e0e41ecca25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c11cefa89d6748769dba4f476cf0cba1"
I20260925 15:01:17.205788 24316 raft_consensus.cc:3037] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:17.205835 24586 raft_consensus.cc:3037] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:17.206496 24316 raft_consensus.cc:2454] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9c60414532419d9c102e0e41ecca25 in term 1.
I20260925 15:01:17.206525 24586 raft_consensus.cc:2454] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc9c60414532419d9c102e0e41ecca25 in term 1.
I20260925 15:01:17.206682 24387 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1; no voters:
I20260925 15:01:17.206812 24646 raft_consensus.cc:2781] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:17.206960 24646 raft_consensus.cc:686] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 1 LEADER]: Becoming Leader. State: Replica: bc9c60414532419d9c102e0e41ecca25, State: Running, Role: LEADER
I20260925 15:01:17.207048 24646 consensus_queue.cc:237] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:17.207893 24174 catalog_manager.cc:5495] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc9c60414532419d9c102e0e41ecca25 (127.8.13.194). New cstate: current_term: 1 leader_uuid: "bc9c60414532419d9c102e0e41ecca25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:17.231822 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.229315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.8.13.194:38507)
0925 15:01:17.229377 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:17.229378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:17.229387 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:01:17.229482 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:17.229483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:17.229502 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:17.229563 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:17.229682 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.230158 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.230917 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:17.231123 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:17.231182 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:17.231382 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:17.231504 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:17.231649 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:17.231670 (+ 21us) server_negotiation.cc:299] Negotiation successful
0925 15:01:17.231694 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260925 15:01:17.233625 24316 raft_consensus.cc:1261] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Refusing update from remote peer bc9c60414532419d9c102e0e41ecca25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:17.233625 24586 raft_consensus.cc:1261] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Refusing update from remote peer bc9c60414532419d9c102e0e41ecca25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:17.233978 24646 consensus_queue.cc:1035] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:17.234141 24649 consensus_queue.cc:1035] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:17.238656 24174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:17.238749 24174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52754:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:17.239471 24174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Sending DeleteTablet for 3 replicas of tablet 03425cb402964a02b82bacfd7b107529
I20260925 15:01:17.239789 24434 tablet_service.cc:1514] Processing DeleteTablet for tablet 03425cb402964a02b82bacfd7b107529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:17 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33934
I20260925 15:01:17.239811 24566 tablet_service.cc:1514] Processing DeleteTablet for tablet 03425cb402964a02b82bacfd7b107529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:17 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:35432
I20260925 15:01:17.239858 24281 tablet_service.cc:1514] Processing DeleteTablet for tablet 03425cb402964a02b82bacfd7b107529 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:17 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:40784
I20260925 15:01:17.239979 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24212
I20260925 15:01:17.240151 24661 tablet_replica.cc:330] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: stopping tablet replica
I20260925 15:01:17.240242 24661 raft_consensus.cc:2229] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:17.240324 24661 raft_consensus.cc:2258] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:17.240453 24662 tablet_replica.cc:330] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: stopping tablet replica
I20260925 15:01:17.240545 24662 raft_consensus.cc:2229] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:17.240638 24662 raft_consensus.cc:2258] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:17.241101 24661 ts_tablet_manager.cc:1905] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:17.241147 24662 ts_tablet_manager.cc:1905] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:17.242901 24661 ts_tablet_manager.cc:1918] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:17.242969 24661 log.cc:1196] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/03425cb402964a02b82bacfd7b107529
I20260925 15:01:17.242985 24662 ts_tablet_manager.cc:1918] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:17.243054 24662 log.cc:1196] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/03425cb402964a02b82bacfd7b107529
I20260925 15:01:17.243324 24662 ts_tablet_manager.cc:1939] T 03425cb402964a02b82bacfd7b107529 P c11cefa89d6748769dba4f476cf0cba1: Deleting consensus metadata
I20260925 15:01:17.243515 24661 ts_tablet_manager.cc:1939] T 03425cb402964a02b82bacfd7b107529 P bc9c60414532419d9c102e0e41ecca25: Deleting consensus metadata
I20260925 15:01:17.243841 24158 catalog_manager.cc:4841] TS c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): tablet 03425cb402964a02b82bacfd7b107529 (table test-table [id=213576ff9e604c8a96cf16ea4e690053]) successfully deleted
I20260925 15:01:17.244232 24161 catalog_manager.cc:4841] TS bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): tablet 03425cb402964a02b82bacfd7b107529 (table test-table [id=213576ff9e604c8a96cf16ea4e690053]) successfully deleted
W20260925 15:01:17.246598 24159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:36597 (error 108)
I20260925 15:01:17.247109 24638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:17.246993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:36597 (local address 127.0.0.1:40794)
0925 15:01:17.247041 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:17.247076 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:36597: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:01:17.247217 24159 catalog_manager.cc:4584] TS aa0ef5fe2fef43e9919487bea2944046 (127.8.13.193:36597): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03425cb402964a02b82bacfd7b107529: Network error: Client connection negotiation failed: client connection to 127.8.13.193:36597: connect: Connection refused (error 111)
I20260925 15:01:17.247876 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24366
I20260925 15:01:17.252789 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24503
I20260925 15:01:17.256763 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24141
I20260925 15:01:37.261827 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--webserver_interface=127.8.13.254
--webserver_port=44791
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37275
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:37.344729 24664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.344966 24664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.344998 24664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.346263 24664 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260925 15:01:37.346306 24664 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260925 15:01:37.346323 24664 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260925 15:01:37.346336 24664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:37.346354 24664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.346371 24664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.346395 24664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:37.346412 24664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:37.347996 24664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37275
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=44791
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:37.348261 24664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.348474 24664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.353557 24670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.353550 24669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.353562 24672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.353595 24664 server_base.cc:1034] running on GCE node
I20260925 15:01:37.354077 24664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.354367 24664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.355511 24664 hybrid_clock.cc:648] HybridClock initialized: now 1790348497355493 us; error 33 us; skew 500 ppm
Sep 25 15:01:37 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.362638 24664 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:01:37.363072 24664 webserver.cc:460] Webserver started at http://127.8.13.254:44791/ using document root <none> and password file <none>
I20260925 15:01:37.363289 24664 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.363339 24664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.364768 24664 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:37.365494 24679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.365700 24664 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:37.365784 24664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "788f5496c2f94536a97e5ed901c0fe98"
format_stamp: "Formatted at 2026-09-25 15:00:55 on dist-test-slave-15hz"
server_key: "66a58a4a4889620b5a124193b48cab3d"
server_key_iv: "b14daf5d228fb19cc1a18885f23c8711"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.366135 24664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.381923 24664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.384102 24664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.384259 24664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.388674 24664 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:37275
I20260925 15:01:37.388720 24731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:37275 every 8 connection(s)
I20260925 15:01:37.389127 24664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:37.390774 24732 sys_catalog.cc:263] Verifying existing consensus state
I20260925 15:01:37.391435 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap starting.
I20260925 15:01:37.397384 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24664
I20260925 15:01:37.397697 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:36597
--local_ip_for_outbound_sockets=127.8.13.193
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=41277
--webserver_interface=127.8.13.193
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:37.400236 24732 log.cc:824] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:37.402486 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:37.402830 24732 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Bootstrap complete.
I20260925 15:01:37.405046 24732 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:37.405293 24732 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Initialized, Role: FOLLOWER
I20260925 15:01:37.405440 24732 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:37.405511 24732 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:37.405556 24732 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:37.405619 24732 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 2 FOLLOWER]: Advancing to term 3
I20260925 15:01:37.406678 24732 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:37.406797 24732 leader_election.cc:302] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788f5496c2f94536a97e5ed901c0fe98; no voters:
I20260925 15:01:37.406976 24732 leader_election.cc:288] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260925 15:01:37.407027 24736 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 3 FOLLOWER]: Leader election won for term 3
I20260925 15:01:37.407138 24736 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [term 3 LEADER]: Becoming Leader. State: Replica: 788f5496c2f94536a97e5ed901c0fe98, State: Running, Role: LEADER
I20260925 15:01:37.407261 24732 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:37.407234 24736 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } }
I20260925 15:01:37.407550 24737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:01:37.407645 24737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:37.407899 24738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788f5496c2f94536a97e5ed901c0fe98. Latest consensus state: current_term: 3 leader_uuid: "788f5496c2f94536a97e5ed901c0fe98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788f5496c2f94536a97e5ed901c0fe98" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37275 } } }
I20260925 15:01:37.408001 24738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:37.407997 24741 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:37.408949 24741 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:37.409282 24741 catalog_manager.cc:1259] Loaded cluster ID: bbf13752eb164f0fab568df01b65b3f0
I20260925 15:01:37.409322 24741 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:37.410553 24741 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:37.410751 24741 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Loaded TSK: 0
I20260925 15:01:37.410802 24741 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 788f5496c2f94536a97e5ed901c0fe98: Loaded TSK: 1
F20260925 15:01:37.411095 24741 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8c2773f9ad google::LogMessage::Fail() at ??:0
@ 0x7f8c2774288a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c2773f4de google::LogMessage::Flush() at ??:0
@ 0x7f8c27740a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c29eac104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8c29ea9d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f8c2f6991cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f8c2f69948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f8c2f699594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f8c2f65fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f8c2f669ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f8c2960d0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f8c29602954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c2a4666db start_thread at ??:0
@ 0x7f8c2623d61f clone at ??:0
W20260925 15:01:37.514457 24734 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.514689 24734 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.514717 24734 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.516026 24734 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.516075 24734 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.516104 24734 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:37.517671 24734 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:36597
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=41277
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24734
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:37.517911 24734 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.518141 24734 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.523114 24773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.523114 24774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.523332 24776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.523623 24734 server_base.cc:1034] running on GCE node
I20260925 15:01:37.523823 24734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.524077 24734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.525219 24734 hybrid_clock.cc:648] HybridClock initialized: now 1790348497525206 us; error 30 us; skew 500 ppm
Sep 25 15:01:37 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.527599 24734 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:01:37.528026 24734 webserver.cc:460] Webserver started at http://127.8.13.193:41277/ using document root <none> and password file <none>
I20260925 15:01:37.528244 24734 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.528294 24734 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.529691 24734 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:37.530416 24783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.530603 24734 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:37.530699 24734 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "aa0ef5fe2fef43e9919487bea2944046"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "e4d2d40e5dc9d33965088506a4449836"
server_key_iv: "f25c73d0f3d26292f45d00e255a4706d"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.531091 24734 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.546615 24734 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.548868 24734 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.549014 24734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.549244 24734 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:37.549682 24790 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260925 15:01:37.551669 24734 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260925 15:01:37.551723 24734 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260925 15:01:37.551764 24734 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260925 15:01:37.552418 24734 ts_tablet_manager.cc:610] Registered 1 tablets
I20260925 15:01:37.552452 24734 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260925 15:01:37.552552 24790 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Bootstrap starting.
I20260925 15:01:37.559582 24734 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:36597
I20260925 15:01:37.559780 24897 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:36597 every 8 connection(s)
I20260925 15:01:37.560040 24734 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:37.563328 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24734
I20260925 15:01:37.563614 24790 log.cc:824] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:37.563664 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:38507
--local_ip_for_outbound_sockets=127.8.13.194
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=34653
--webserver_interface=127.8.13.194
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:01:37.566412 24790 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260925 15:01:37.566767 24790 tablet_bootstrap.cc:492] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Bootstrap complete.
I20260925 15:01:37.567898 24790 ts_tablet_manager.cc:1397] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260925 15:01:37.569479 24790 raft_consensus.cc:348] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:37.569832 24790 raft_consensus.cc:729] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa0ef5fe2fef43e9919487bea2944046, State: Initialized, Role: FOLLOWER
I20260925 15:01:37.570012 24790 consensus_queue.cc:260] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:37.570307 24790 ts_tablet_manager.cc:1428] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20260925 15:01:37.631292 24900 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:01:37.560705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:48645)
0925 15:01:37.560874 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.560920 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:37.561407 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:37.631087 (+ 69680us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
W20260925 15:01:37.631613 24898 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:37275 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:37275: Client connection negotiation failed: client connection to 127.8.13.254:37275: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260925 15:01:37.632038 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.631915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:46675)
0925 15:01:37.631969 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.632012 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260925 15:01:37.632372 24900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.632281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:57623)
0925 15:01:37.632325 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.632349 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260925 15:01:37.632489 24898 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260925 15:01:37.648795 24902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.649039 24902 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.649075 24902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.650271 24902 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.650316 24902 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.650360 24902 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:37.651925 24902 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:38507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=34653
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24902
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:37.652170 24902 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.652391 24902 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.657348 24909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.657379 24908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.657397 24911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.657639 24902 server_base.cc:1034] running on GCE node
I20260925 15:01:37.657845 24902 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.658097 24902 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.659241 24902 hybrid_clock.cc:648] HybridClock initialized: now 1790348497659226 us; error 28 us; skew 500 ppm
Sep 25 15:01:37 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.661645 24902 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:01:37.662086 24902 webserver.cc:460] Webserver started at http://127.8.13.194:34653/ using document root <none> and password file <none>
I20260925 15:01:37.662302 24902 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.662353 24902 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.663721 24902 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:01:37.664402 24918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.664583 24902 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:01:37.664661 24902 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "bc9c60414532419d9c102e0e41ecca25"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "66f982fd23cf4f93e84af27ae214d498"
server_key_iv: "3f1ba13a03295d9e053896f6c52cf80b"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.665010 24902 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.672358 24902 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.674453 24902 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.674587 24902 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.674808 24902 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:37.675086 24902 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:37.675118 24902 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.675145 24902 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:37.675169 24902 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.680920 24902 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:38507
I20260925 15:01:37.680977 25031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:38507 every 8 connection(s)
I20260925 15:01:37.681310 24902 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:37.682329 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.681950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:44617)
0925 15:01:37.682100 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.682182 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":75,"threads_started":1}
W20260925 15:01:37.682709 25032 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:37275 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:37275: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
I20260925 15:01:37.682964 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.682875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:37849)
0925 15:01:37.682916 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.682940 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 15:01:37.683194 25034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.683123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:34323)
0925 15:01:37.683156 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.683176 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 15:01:37.683280 25032 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:01:37.689754 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 24902
I20260925 15:01:37.690040 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:44239
--local_ip_for_outbound_sockets=127.8.13.195
--tserver_master_addrs=127.8.13.254:37275
--webserver_port=40631
--webserver_interface=127.8.13.195
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33677
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:37.770188 25035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:37.770424 25035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:37.770458 25035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:37.771677 25035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:37.771723 25035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:37.771759 25035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:37.773252 25035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33677
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:44239
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=40631
--tserver_master_addrs=127.8.13.254:37275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:37.773497 25035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:37.773723 25035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:37.778748 25040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.778846 25035 server_base.cc:1034] running on GCE node
W20260925 15:01:37.778769 25041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:37.778967 25043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:37.779165 25035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:37.779419 25035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:37.780601 25035 hybrid_clock.cc:648] HybridClock initialized: now 1790348497780598 us; error 39 us; skew 500 ppm
Sep 25 15:01:37 dist-test-slave-15hz krb5kdc[23594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:37.783089 25035 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:01:37.783634 25035 webserver.cc:460] Webserver started at http://127.8.13.195:40631/ using document root <none> and password file <none>
I20260925 15:01:37.783843 25035 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:37.783905 25035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:37.785291 25035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:37.785902 25050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.786068 25035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:37.786149 25035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "c11cefa89d6748769dba4f476cf0cba1"
format_stamp: "Formatted at 2026-09-25 15:00:56 on dist-test-slave-15hz"
server_key: "0e1376c01387f1cb9995c5b05a56cd8b"
server_key_iv: "4da9231ba7c265a7c76997bc6ceebcd9"
server_key_version: "encryptionkey@0"
I20260925 15:01:37.786518 25035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:37.809325 25035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:37.811420 25035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:37.811601 25035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:37.811839 25035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:37.812152 25035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:37.812186 25035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.812217 25035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:37.812234 25035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:37.817633 25035 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:44239
I20260925 15:01:37.817694 25163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:44239 every 8 connection(s)
I20260925 15:01:37.818043 25035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:37.819100 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.818725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:50459)
0925 15:01:37.818880 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.818954 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":77,"threads_started":1}
W20260925 15:01:37.819559 25164 heartbeater.cc:644] Failed to heartbeat to 127.8.13.254:37275 (0 consecutive failures): Network error: Failed to ping master at 127.8.13.254:37275: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
I20260925 15:01:37.819864 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.819778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:46405)
0925 15:01:37.819818 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.819846 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260925 15:01:37.820106 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:37.820051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:54739)
0925 15:01:37.820074 (+ 23us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:37.820093 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260925 15:01:37.820210 25164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260925 15:01:37.825362 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25035
I20260925 15:01:38.633049 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.632829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:43005)
0925 15:01:38.632962 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.633000 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260925 15:01:38.683815 25169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.683612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:52765)
0925 15:01:38.683726 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.683757 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260925 15:01:38.755156 25170 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:38.755291 25170 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:38.755667 25170 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
Sep 25 15:01:38 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.195@KRBTEST.COM
Sep 25 15:01:38 dist-test-slave-15hz krb5kdc[23594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348497, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.194@KRBTEST.COM
I20260925 15:01:38.765705 25173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.755953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:41175 (local address 127.8.13.195:44239)
0925 15:01:38.756104 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:38.756116 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:38.756138 (+ 22us) server_negotiation.cc:407] Connection header received
0925 15:01:38.756573 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:38.756576 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:38.756637 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:38.756764 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:38.757642 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.758507 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.759556 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.759956 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.760053 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:38.762304 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:38.762309 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:38.762313 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:38.762327 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:38.764060 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764561 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:38.764565 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:38.764566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:38.764657 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764891 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:38.764892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:38.764893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:38.765124 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:38.765205 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:38.765479 (+ 274us) server_negotiation.cc:299] Negotiation successful
0925 15:01:38.765604 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":52,"threads_started":1}
I20260925 15:01:38.765719 25172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.755947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:32951 (local address 127.8.13.194:38507)
0925 15:01:38.756116 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:38.756124 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:38.756152 (+ 28us) server_negotiation.cc:407] Connection header received
0925 15:01:38.756608 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:38.756611 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:38.756647 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:38.756765 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:38.757634 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.758504 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.759561 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.759957 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.760053 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:38.762630 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:38.762634 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:38.762634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:38.762645 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:38.764102 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764562 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:38.764565 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:38.764566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:38.764658 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764886 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:38.764887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:38.764887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:38.765123 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:38.765201 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:38.765480 (+ 279us) server_negotiation.cc:299] Negotiation successful
0925 15:01:38.765605 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":65,"threads_started":1}
I20260925 15:01:38.765710 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.755855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:44239 (local address 127.8.13.193:41175)
0925 15:01:38.755964 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.755976 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:38.756037 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:38.756831 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:38.756834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:38.757275 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:38.757515 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:38.757521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.758618 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.758619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:38.759451 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:38.759453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.760038 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.760039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:38.760152 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:38.760737 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:38.760741 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:38.762179 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:38.764141 (+ 1962us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764149 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:38.764152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:38.764484 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:38.764719 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764719 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:38.764720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:38.764832 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:38.765214 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:38.765215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:38.765330 (+ 115us) client_negotiation.cc:769] Sending connection context
0925 15:01:38.765417 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:01:38.765618 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":54}
I20260925 15:01:38.766324 25171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.755862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:38507 (local address 127.8.13.193:32951)
0925 15:01:38.756007 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.756017 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:38.756071 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:38.756831 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:38.756834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:38.757264 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:38.757515 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:38.757521 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.758603 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.758604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:38.759453 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:38.759455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:38.760040 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:38.760041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:38.760152 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:38.760687 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:38.760695 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:38.762535 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:38.764187 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764188 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:38.764188 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:38.764486 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:38.764720 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:38.764720 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:38.764721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:38.764829 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:38.765191 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:38.765208 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:38.765330 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 15:01:38.765417 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:01:38.765697 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":190,"thread_start_us":70,"threads_started":1}
W20260925 15:01:38.766609 24784 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
W20260925 15:01:38.766935 24787 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
I20260925 15:01:38.766985 24787 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046; no voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1
I20260925 15:01:38.767081 25170 raft_consensus.cc:2726] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:01:38.820730 25174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:38.820499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:50627)
0925 15:01:38.820637 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:38.820688 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
I20260925 15:01:39.633821 25175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.633590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:33763)
0925 15:01:39.633719 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.633770 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":64,"threads_started":1}
I20260925 15:01:39.684479 25176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.684236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:39925)
0925 15:01:39.684386 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.684431 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260925 15:01:39.821436 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:39.821208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:35203)
0925 15:01:39.821345 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:39.821390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260925 15:01:40.634608 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.634340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:39451)
0925 15:01:40.634507 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.634555 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":78,"threads_started":1}
I20260925 15:01:40.685343 25179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.685069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:46861)
0925 15:01:40.685242 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.685291 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260925 15:01:40.822224 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:40.821964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:47321)
0925 15:01:40.822129 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:40.822176 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":68,"threads_started":1}
I20260925 15:01:40.840873 25181 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:40.840982 25181 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:40.841161 25181 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
W20260925 15:01:40.841622 24784 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
W20260925 15:01:40.841692 24787 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
I20260925 15:01:40.841786 24787 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046; no voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1
I20260925 15:01:40.841871 25181 raft_consensus.cc:2726] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:01:41.635383 25182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.635100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:37263)
0925 15:01:41.635291 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.635337 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":82,"threads_started":1}
I20260925 15:01:41.686088 25183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.685860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:39701)
0925 15:01:41.685987 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.686040 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260925 15:01:41.823082 25184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:41.822843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:42511)
0925 15:01:41.822992 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:41.823034 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":82,"threads_started":1}
I20260925 15:01:42.636314 25185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.635982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:54713)
0925 15:01:42.636219 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.636265 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":89,"threads_started":1}
I20260925 15:01:42.686939 25186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.686655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:45687)
0925 15:01:42.686838 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.686887 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":75,"threads_started":1}
I20260925 15:01:42.823966 25187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:42.823705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:60797)
0925 15:01:42.823872 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:42.823918 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":62,"threads_started":1}
I20260925 15:01:43.637102 25188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.636846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:47321)
0925 15:01:43.637016 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.637054 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":81,"threads_started":1}
I20260925 15:01:43.687858 25189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.687579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:56733)
0925 15:01:43.687759 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.687805 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":91,"threads_started":1}
I20260925 15:01:43.824855 25190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:43.824551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:57621)
0925 15:01:43.824747 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:43.824806 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":98,"threads_started":1}
I20260925 15:01:43.926085 25191 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:43.926194 25191 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:43.926353 25191 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
W20260925 15:01:43.926784 24787 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
W20260925 15:01:43.926889 24784 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
I20260925 15:01:43.926944 24784 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046; no voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1
I20260925 15:01:43.927008 25191 raft_consensus.cc:2726] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:01:44.637976 25192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.637701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:55591)
0925 15:01:44.637875 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.637926 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":64,"threads_started":1}
I20260925 15:01:44.688681 25193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.688405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:54001)
0925 15:01:44.688587 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.688632 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
I20260925 15:01:44.825601 25194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:44.825372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:41371)
0925 15:01:44.825513 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:44.825554 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260925 15:01:45.638803 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.638529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:46427)
0925 15:01:45.638704 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.638747 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":89,"threads_started":1}
I20260925 15:01:45.689407 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.689188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:49301)
0925 15:01:45.689311 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.689358 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260925 15:01:45.826418 25197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:45.826150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:53507)
0925 15:01:45.826318 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:45.826366 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260925 15:01:46.639597 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.639314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:46409)
0925 15:01:46.639468 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.639516 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260925 15:01:46.690131 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.689831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:54045)
0925 15:01:46.690035 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.690082 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":63,"threads_started":1}
I20260925 15:01:46.827152 25200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:46.826932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:53833)
0925 15:01:46.827064 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:46.827106 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260925 15:01:47.640280 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.640037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:53127)
0925 15:01:47.640177 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.640231 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":60,"threads_started":1}
I20260925 15:01:47.690812 25202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.690601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:53949)
0925 15:01:47.690714 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.690756 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":61,"threads_started":1}
I20260925 15:01:47.827886 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:47.827649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:40891)
0925 15:01:47.827794 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:47.827839 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":60,"threads_started":1}
I20260925 15:01:48.485852 25204 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:48.485960 25204 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:48.486155 25204 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
W20260925 15:01:48.486505 24787 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
W20260925 15:01:48.486593 24784 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
I20260925 15:01:48.486662 24784 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046; no voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1
I20260925 15:01:48.486721 25204 raft_consensus.cc:2726] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:01:48.640990 25205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.640766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:55957)
0925 15:01:48.640896 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.640940 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":63,"threads_started":1}
I20260925 15:01:48.691452 25206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.691240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:48435)
0925 15:01:48.691365 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.691405 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260925 15:01:48.828574 25207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:48.828352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:50521)
0925 15:01:48.828486 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:48.828530 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260925 15:01:49.641696 25208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.641456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:57061)
0925 15:01:49.641603 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.641647 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260925 15:01:49.692130 25209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.691897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:48801)
0925 15:01:49.692039 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.692082 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260925 15:01:49.829362 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:49.829057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:45405)
0925 15:01:49.829250 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:49.829299 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":103,"threads_started":1}
I20260925 15:01:50.063604 25211 raft_consensus.cc:482] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:50.063699 25211 raft_consensus.cc:504] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc9c60414532419d9c102e0e41ecca25" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 38507 } } peers { permanent_uuid: "c11cefa89d6748769dba4f476cf0cba1" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 44239 } } peers { permanent_uuid: "aa0ef5fe2fef43e9919487bea2944046" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 36597 } }
I20260925 15:01:50.063894 25211 leader_election.cc:288] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507), c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239)
W20260925 15:01:50.064276 24787 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer bc9c60414532419d9c102e0e41ecca25 (127.8.13.194:38507): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
W20260925 15:01:50.064357 24784 leader_election.cc:341] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c11cefa89d6748769dba4f476cf0cba1 (127.8.13.195:44239): Not found: Tablet not found: 03425cb402964a02b82bacfd7b107529
I20260925 15:01:50.064420 24784 leader_election.cc:302] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: aa0ef5fe2fef43e9919487bea2944046; no voters: bc9c60414532419d9c102e0e41ecca25, c11cefa89d6748769dba4f476cf0cba1
I20260925 15:01:50.064491 25211 raft_consensus.cc:2726] T 03425cb402964a02b82bacfd7b107529 P aa0ef5fe2fef43e9919487bea2944046 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260925 15:01:50.642483 25212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.642258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:38051)
0925 15:01:50.642387 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.642434 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260925 15:01:50.692853 25213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.692603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:49019)
0925 15:01:50.692756 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.692804 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":61,"threads_started":1}
I20260925 15:01:50.830135 25214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:50.829890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:45415)
0925 15:01:50.830040 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:50.830086 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260925 15:01:51.643224 25215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.642984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:35699)
0925 15:01:51.643124 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.643173 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":65,"threads_started":1}
I20260925 15:01:51.693666 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.693405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:37943)
0925 15:01:51.693570 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.693617 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":69,"threads_started":1}
I20260925 15:01:51.834857 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:51.834588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:57783)
0925 15:01:51.834757 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:51.834797 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":65,"threads_started":1}
I20260925 15:01:52.644035 25218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.643816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.193:44887)
0925 15:01:52.643947 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.643985 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260925 15:01:52.694471 25219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.694244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.194:37369)
0925 15:01:52.694380 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.694425 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260925 15:01:52.832744 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24734
I20260925 15:01:52.835711 25220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:52.835438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37275 (local address 127.8.13.195:59611)
0925 15:01:52.835608 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:52.835655 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.254:37275: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":53,"threads_started":1}
I20260925 15:01:52.837345 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 24902
I20260925 15:01:52.841161 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25035
2026-09-25T15:01:52Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58159 ms)
[----------] 30 tests from SecurityITest (228424 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:52 dist-test-slave-15hz krb5kdc[25223](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:52 dist-test-slave-15hz krb5kdc[25223](info): set up 2 sockets
Sep 25 15:01:52 dist-test-slave-15hz krb5kdc[25223](info): commencing operation
krb5kdc: starting...
W20260925 15:01:53.823267 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.963s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:01:53 dist-test-slave-15hz krb5kdc[25223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:01:53.938858 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44927
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44927
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:54.017400 25236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:54.017604 25236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:54.017634 25236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:54.018771 25236 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:54.018815 25236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:54.018828 25236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:54.018839 25236 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:54.018849 25236 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:54.020329 25236 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44927
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:54.020565 25236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:54.020788 25236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:54.025950 25244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.026019 25242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.025950 25241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.025979 25236 server_base.cc:1034] running on GCE node
I20260925 15:01:54.026436 25236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:54.026743 25236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:54.027936 25236 hybrid_clock.cc:648] HybridClock initialized: now 1790348514027918 us; error 38 us; skew 500 ppm
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:54.030596 25236 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:01:54.031097 25236 webserver.cc:460] Webserver started at http://127.8.13.254:40439/ using document root <none> and password file <none>
I20260925 15:01:54.031307 25236 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:54.031356 25236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:54.031481 25236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:54.032573 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "08cfafe9a29744c5ad30c3af07ae8c79"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5abb6c6cdb7538603dddc515d497c722"
server_key_iv: "c58f08bc46d866faefa1df18df7dab68"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.032945 25236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "08cfafe9a29744c5ad30c3af07ae8c79"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5abb6c6cdb7538603dddc515d497c722"
server_key_iv: "c58f08bc46d866faefa1df18df7dab68"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.034221 25236 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:54.035005 25251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.035183 25236 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:54.035277 25236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "08cfafe9a29744c5ad30c3af07ae8c79"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5abb6c6cdb7538603dddc515d497c722"
server_key_iv: "c58f08bc46d866faefa1df18df7dab68"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.035351 25236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:54.044039 25236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:54.045746 25236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:54.045871 25236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:54.049413 25236 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44927
I20260925 15:01:54.049446 25303 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44927 every 8 connection(s)
I20260925 15:01:54.049765 25236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:54.050570 25304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:54.052781 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: Bootstrap starting.
I20260925 15:01:54.053303 25304 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:54.053303 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25236
I20260925 15:01:54.053390 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:54.053556 8247 external_mini_cluster.cc:1428] Setting key 70914646f15f124a17f7ef3ffebded08
I20260925 15:01:54.053567 25304 log.cc:824] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:54.054306 25304 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: No bootstrap required, opened a new log
I20260925 15:01:54.056874 25304 raft_consensus.cc:348] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } }
I20260925 15:01:54.057049 25304 raft_consensus.cc:374] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:54.057081 25304 raft_consensus.cc:729] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 08cfafe9a29744c5ad30c3af07ae8c79, State: Initialized, Role: FOLLOWER
I20260925 15:01:54.057204 25304 consensus_queue.cc:260] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } }
I20260925 15:01:54.057299 25304 raft_consensus.cc:388] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:54.057330 25304 raft_consensus.cc:482] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:54.057367 25304 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:54.058061 25304 raft_consensus.cc:504] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } }
I20260925 15:01:54.058171 25304 leader_election.cc:302] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 08cfafe9a29744c5ad30c3af07ae8c79; no voters:
I20260925 15:01:54.058342 25304 leader_election.cc:288] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:54.058368 25309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:54.058485 25309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [term 1 LEADER]: Becoming Leader. State: Replica: 08cfafe9a29744c5ad30c3af07ae8c79, State: Running, Role: LEADER
I20260925 15:01:54.058606 25304 sys_catalog.cc:564] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:54.058625 25309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } }
I20260925 15:01:54.058939 25309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 08cfafe9a29744c5ad30c3af07ae8c79. Latest consensus state: current_term: 1 leader_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } } }
I20260925 15:01:54.059015 25310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "08cfafe9a29744c5ad30c3af07ae8c79" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44927 } } }
I20260925 15:01:54.059188 25310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:54.059078 25309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:54.059398 25318 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:54.059963 25318 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348513, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:54.062834 25318 catalog_manager.cc:1347] Generated new cluster ID: ea43d07aad524134a286e20cf34d6baa
I20260925 15:01:54.062932 25318 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:54.066401 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.054074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49966 (local address 127.8.13.254:44927)
0925 15:01:54.054270 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.054279 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.054312 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:01:54.054816 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.054825 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.054890 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.055039 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.055689 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.057094 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.058129 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.058732 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.058885 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.062301 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.062306 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.062306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:54.062319 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.064513 (+ 2194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.064991 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.064995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.064995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.065115 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.065361 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.065362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.065362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.065632 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.065791 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.065915 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.066052 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":97,"threads_started":1}
I20260925 15:01:54.072067 25318 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:54.072582 25318 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:54.074831 25318 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: Generated new TSK 0
I20260925 15:01:54.074987 25318 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:01:54.104468 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:54.183964 25330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:54.184194 25330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:54.184219 25330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:54.185425 25330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:54.185474 25330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:54.185504 25330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:54.187007 25330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:54.187256 25330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:54.187479 25330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:54.192749 25336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.192766 25338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.192790 25330 server_base.cc:1034] running on GCE node
W20260925 15:01:54.192755 25335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.193157 25330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:54.193385 25330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:54.194532 25330 hybrid_clock.cc:648] HybridClock initialized: now 1790348514194515 us; error 28 us; skew 500 ppm
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:54.197186 25330 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:01:54.197649 25330 webserver.cc:460] Webserver started at http://127.8.13.193:36313/ using document root <none> and password file <none>
I20260925 15:01:54.197835 25330 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:54.197875 25330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:54.197969 25330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:54.199028 25330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "908cdccc69d4449aa6c9c758d47fe5b7"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5b008b45992e8e01d60fe0904eb8bcba"
server_key_iv: "3d3076fd057ca0561f1becd13ce42900"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.199328 25330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "908cdccc69d4449aa6c9c758d47fe5b7"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5b008b45992e8e01d60fe0904eb8bcba"
server_key_iv: "3d3076fd057ca0561f1becd13ce42900"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.200567 25330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:54.201272 25345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.201400 25330 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:54.201480 25330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "908cdccc69d4449aa6c9c758d47fe5b7"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "5b008b45992e8e01d60fe0904eb8bcba"
server_key_iv: "3d3076fd057ca0561f1becd13ce42900"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.201547 25330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:54.227020 25330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:54.228968 25330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:54.229125 25330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:54.229346 25330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:54.229651 25330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:54.229683 25330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.229704 25330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:54.229727 25330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.235502 25330 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:41475
I20260925 15:01:54.235591 25458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:41475 every 8 connection(s)
I20260925 15:01:54.235937 25330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:54.239274 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25330
I20260925 15:01:54.239384 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:54.239604 8247 external_mini_cluster.cc:1428] Setting key 712aa16fb304a42bfc25caba64929690
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:54.245963 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.236663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:56297 (local address 127.8.13.254:44927)
0925 15:01:54.236742 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.236745 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.236903 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:01:54.237438 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.237440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.237452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.237502 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.238488 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.238996 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.240330 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.240524 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.240578 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.242861 (+ 2283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.242862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.242862 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:54.242870 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.244408 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.244917 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.244918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.244919 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.244969 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.245284 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.245285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.245285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.245411 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.245468 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.245816 (+ 348us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.245893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260925 15:01:54.246234 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.236587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44927 (local address 127.8.13.193:56297)
0925 15:01:54.236773 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.236822 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.237337 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.237587 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.237591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.238195 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:54.238395 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.238400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.239084 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.239085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.240254 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.240257 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.240601 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.240602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.240748 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.241225 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:54.241235 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:54.242756 (+ 1521us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:54.244501 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.244512 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.244515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.244846 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.245031 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.245031 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.245032 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.245185 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.245475 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:54.245481 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:54.245676 (+ 195us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.245795 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.245981 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260925 15:01:54.246785 25459 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44927
I20260925 15:01:54.246891 25459 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:54.247094 25459 heartbeater.cc:511] Master 127.8.13.254:44927 requested a full tablet report, sending...
I20260925 15:01:54.247618 25268 ts_manager.cc:194] Registered new tserver with Master: 908cdccc69d4449aa6c9c758d47fe5b7 (127.8.13.193:41475)
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 15:01:54.248265 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:56297
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:01:54.261929 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:54.340195 25464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:54.340409 25464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:54.340441 25464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:54.341631 25464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:54.341672 25464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:54.341707 25464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:54.343133 25464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:54.343365 25464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:54.343603 25464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:54.348451 25469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.348462 25470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.348475 25472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.348721 25464 server_base.cc:1034] running on GCE node
I20260925 15:01:54.348902 25464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:54.349148 25464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:54.350308 25464 hybrid_clock.cc:648] HybridClock initialized: now 1790348514350295 us; error 29 us; skew 500 ppm
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:54.352736 25464 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:01:54.353193 25464 webserver.cc:460] Webserver started at http://127.8.13.194:43349/ using document root <none> and password file <none>
I20260925 15:01:54.353394 25464 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:54.353443 25464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:54.353583 25464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:54.354614 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "a8bd5d7a49274738bfa387d9d0441bd3"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "900c7c7f014f46f0dd0c60cc486faf54"
server_key_iv: "88ea02fd7107fea6894eb52791d2ec9e"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.354950 25464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "a8bd5d7a49274738bfa387d9d0441bd3"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "900c7c7f014f46f0dd0c60cc486faf54"
server_key_iv: "88ea02fd7107fea6894eb52791d2ec9e"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.356298 25464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:01:54.356950 25479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.357131 25464 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:54.357214 25464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "a8bd5d7a49274738bfa387d9d0441bd3"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "900c7c7f014f46f0dd0c60cc486faf54"
server_key_iv: "88ea02fd7107fea6894eb52791d2ec9e"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.357278 25464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:54.384843 25464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:54.386690 25464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:54.386842 25464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:54.387081 25464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:54.387400 25464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:54.387437 25464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.387471 25464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:54.387495 25464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.393528 25464 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:43807
I20260925 15:01:54.393569 25592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:43807 every 8 connection(s)
I20260925 15:01:54.393910 25464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:54.396521 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25464
I20260925 15:01:54.396639 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:54.396842 8247 external_mini_cluster.cc:1428] Setting key ba2656552b656cdaf7264ae66245857e
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:54.404438 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.394634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:43837 (local address 127.8.13.254:44927)
0925 15:01:54.394678 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.394680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.394870 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:01:54.395395 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.395396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.395407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.395472 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.396554 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.397232 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.398328 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.398607 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.398682 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.401193 (+ 2511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.401195 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.401195 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:54.401204 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.402665 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.403207 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.403208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.403209 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.403275 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.403639 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.403640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.403640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.403773 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.403842 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.404287 (+ 445us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.404366 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:01:54.404687 25595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.394601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44927 (local address 127.8.13.194:43837)
0925 15:01:54.394758 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.394801 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.395283 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.395616 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.395620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.396211 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:54.396434 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.396440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.397323 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.397324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.398237 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.398241 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.398685 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.398686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.398836 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.399369 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:54.399379 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:54.401054 (+ 1675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:54.402758 (+ 1704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.402769 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.402772 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.403102 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.403347 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.403348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.403348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.403506 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.403852 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:54.403861 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:54.404077 (+ 216us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.404231 (+ 154us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.404435 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 15:01:54.405303 25593 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44927
I20260925 15:01:54.405412 25593 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:54.405596 25593 heartbeater.cc:511] Master 127.8.13.254:44927 requested a full tablet report, sending...
I20260925 15:01:54.405973 25268 ts_manager.cc:194] Registered new tserver with Master: a8bd5d7a49274738bfa387d9d0441bd3 (127.8.13.194:43807)
I20260925 15:01:54.406345 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:43837
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:01:54.421923 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:54.503417 25598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:54.503640 25598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:54.503662 25598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:54.504907 25598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:54.504951 25598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:54.504977 25598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:54.506441 25598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:44927
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:54.506664 25598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:54.506876 25598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:54.511552 25604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.511723 25598 server_base.cc:1034] running on GCE node
W20260925 15:01:54.511584 25603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.511561 25606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.512028 25598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:54.512276 25598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:54.513412 25598 hybrid_clock.cc:648] HybridClock initialized: now 1790348514513394 us; error 36 us; skew 500 ppm
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:54.515758 25598 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:01:54.516213 25598 webserver.cc:460] Webserver started at http://127.8.13.195:32817/ using document root <none> and password file <none>
I20260925 15:01:54.516407 25598 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:54.516454 25598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:54.516582 25598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:54.517617 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "44e31fc8cfc8410cbec4f3c2e6a87003"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "810558d75ef8b6e167ae5d2c04a36c3a"
server_key_iv: "1cfac6cd9e552387f0df3c042fef7b9d"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.517958 25598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "44e31fc8cfc8410cbec4f3c2e6a87003"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "810558d75ef8b6e167ae5d2c04a36c3a"
server_key_iv: "1cfac6cd9e552387f0df3c042fef7b9d"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.519269 25598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:54.520008 25613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.520170 25598 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.520252 25598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "44e31fc8cfc8410cbec4f3c2e6a87003"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "810558d75ef8b6e167ae5d2c04a36c3a"
server_key_iv: "1cfac6cd9e552387f0df3c042fef7b9d"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.520323 25598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:54.546693 25598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:54.548941 25598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:54.549089 25598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:54.549325 25598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:54.549614 25598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:54.549647 25598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.549677 25598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:54.549695 25598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.555409 25598 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:43535
I20260925 15:01:54.555459 25726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:43535 every 8 connection(s)
I20260925 15:01:54.555801 25598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:54.556927 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25598
I20260925 15:01:54.557041 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:54.557247 8247 external_mini_cluster.cc:1428] Setting key ab2f72fd74d29ccb4d8477062e894610
Sep 25 15:01:54 dist-test-slave-15hz krb5kdc[25223](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348514, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:54.566427 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.556487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:55337 (local address 127.8.13.254:44927)
0925 15:01:54.556528 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.556531 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.556721 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:01:54.557387 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.557388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.557402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.557469 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.558603 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.559180 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.560383 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.560574 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.560630 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.562659 (+ 2029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.562660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.562661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:54.562669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.564767 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.565282 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.565284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.565284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.565350 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.565622 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.565623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.565623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.565758 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.565818 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.566213 (+ 395us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.566315 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:01:54.566596 25729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.556464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:44927 (local address 127.8.13.195:55337)
0925 15:01:54.556601 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.556639 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.557277 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.557542 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.557546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.558312 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:54.558518 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.558523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.559253 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.559254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.560314 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.560317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.560636 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.560637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.560733 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.561208 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:54.561218 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:54.562575 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:54.564859 (+ 2284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.564870 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.564874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.565191 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.565417 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:54.565417 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:54.565418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:54.565560 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:54.565817 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:54.565824 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:54.566012 (+ 188us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.566157 (+ 145us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.566347 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260925 15:01:54.567369 25727 heartbeater.cc:347] Connected to a master server at 127.8.13.254:44927
I20260925 15:01:54.567678 25727 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:54.567891 25727 heartbeater.cc:511] Master 127.8.13.254:44927 requested a full tablet report, sending...
I20260925 15:01:54.568399 25268 ts_manager.cc:194] Registered new tserver with Master: 44e31fc8cfc8410cbec4f3c2e6a87003 (127.8.13.195:43535)
I20260925 15:01:54.568866 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:54.568912 25268 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:55337
I20260925 15:01:54.578266 25308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.570465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49972 (local address 127.8.13.254:44927)
0925 15:01:54.570637 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.570639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.570650 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:01:54.570673 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.570673 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.570685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.570745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.571354 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.572048 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.573196 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.573450 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.573507 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.575033 (+ 1526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.575034 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.575034 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:54.575041 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.576966 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.577371 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.577373 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.577373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.577455 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:54.577658 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:54.577659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:54.577660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:54.577895 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.578009 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.578118 (+ 109us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.578194 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260925 15:01:54.580034 25268 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49972:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:54.582094 25268 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:54.589257 25738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43535 (local address 127.0.0.1:45180)
0925 15:01:54.584679 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.584725 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.584807 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.585185 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.585188 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.585190 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:54.585272 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.585276 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.586663 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.586664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.587966 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.587969 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.588929 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.588930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.589031 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.589048 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.589112 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.589170 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"thread_start_us":58,"threads_started":1}
I20260925 15:01:54.589560 25742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41475 (local address 127.0.0.1:36242)
0925 15:01:54.585269 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.585279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.585328 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.585631 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.585631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.585633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:54.585680 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.585682 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.587100 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.587101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.588448 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.588449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.589349 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.589350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.589410 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.589414 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.589441 (+ 27us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.589482 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":49,"thread_start_us":67,"threads_started":1}
I20260925 15:01:54.589582 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.8.13.195:43535)
0925 15:01:54.585009 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.585015 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.585025 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:54.585069 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.585071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.585087 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.585163 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:54.585578 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.586567 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.588059 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.588756 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.588826 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.588891 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.589452 (+ 561us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.589499 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":88,"threads_started":1}
I20260925 15:01:54.589706 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55938 (local address 127.8.13.194:43807)
0925 15:01:54.585446 (+ 695us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.585449 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.585459 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:54.585495 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.585497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.585511 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.585786 (+ 275us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:54.585902 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.586990 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.588536 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.589264 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.589319 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.589375 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.589595 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.589636 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":84,"threads_started":1}
I20260925 15:01:54.589789 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:43807 (local address 127.0.0.1:55938)
0925 15:01:54.584906 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.584914 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.584947 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.585796 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.585797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.585800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:54.585830 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.585832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.587095 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.587096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.588446 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.588448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.589403 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.589404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.589452 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.589455 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.589493 (+ 38us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.589524 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260925 15:01:54.590327 25661 tablet_service.cc:1467] Processing CreateTablet for tablet 31ef7871a52547afb73187698affb045 (DEFAULT_TABLE table=test-table [id=c921e5e6343e44b2bba0374e8bcd616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:54.590473 25527 tablet_service.cc:1467] Processing CreateTablet for tablet 31ef7871a52547afb73187698affb045 (DEFAULT_TABLE table=test-table [id=c921e5e6343e44b2bba0374e8bcd616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:54.590698 25661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ef7871a52547afb73187698affb045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:54.590734 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ef7871a52547afb73187698affb045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:54.591871 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.584969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36242 (local address 127.8.13.193:41475)
0925 15:01:54.585145 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.585149 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.585323 (+ 174us) server_negotiation.cc:407] Connection header received
0925 15:01:54.585400 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.585403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.585418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.585489 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:54.585868 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.586990 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.588536 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.589253 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.591631 (+ 2378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.591709 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.591745 (+ 36us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.591793 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":83,"threads_started":1}
I20260925 15:01:54.593968 25745 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Bootstrap starting.
I20260925 15:01:54.594586 25393 tablet_service.cc:1467] Processing CreateTablet for tablet 31ef7871a52547afb73187698affb045 (DEFAULT_TABLE table=test-table [id=c921e5e6343e44b2bba0374e8bcd616a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:54.594708 25745 tablet_bootstrap.cc:654] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:54.594894 25393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31ef7871a52547afb73187698affb045. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:54.595249 25745 log.cc:824] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:54.595633 25744 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Bootstrap starting.
I20260925 15:01:54.596081 25745 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: No bootstrap required, opened a new log
I20260925 15:01:54.596213 25745 ts_tablet_manager.cc:1397] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:54.596307 25744 tablet_bootstrap.cc:654] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:54.596848 25744 log.cc:824] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:54.597533 25744 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: No bootstrap required, opened a new log
I20260925 15:01:54.597647 25744 ts_tablet_manager.cc:1397] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:54.598030 25748 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Bootstrap starting.
I20260925 15:01:54.598639 25748 tablet_bootstrap.cc:654] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:54.598664 25745 raft_consensus.cc:348] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.598829 25745 raft_consensus.cc:374] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:54.598869 25745 raft_consensus.cc:729] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8bd5d7a49274738bfa387d9d0441bd3, State: Initialized, Role: FOLLOWER
I20260925 15:01:54.598986 25745 consensus_queue.cc:260] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.599170 25748 log.cc:824] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:54.599258 25745 ts_tablet_manager.cc:1428] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:01:54.599270 25593 heartbeater.cc:503] Master 127.8.13.254:44927 was elected leader, sending a full tablet report...
I20260925 15:01:54.599954 25748 tablet_bootstrap.cc:492] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: No bootstrap required, opened a new log
I20260925 15:01:54.600028 25748 ts_tablet_manager.cc:1397] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:54.600133 25744 raft_consensus.cc:348] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.600299 25744 raft_consensus.cc:374] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:54.600338 25744 raft_consensus.cc:729] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44e31fc8cfc8410cbec4f3c2e6a87003, State: Initialized, Role: FOLLOWER
I20260925 15:01:54.600449 25744 consensus_queue.cc:260] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.600716 25744 ts_tablet_manager.cc:1428] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:54.600744 25727 heartbeater.cc:503] Master 127.8.13.254:44927 was elected leader, sending a full tablet report...
I20260925 15:01:54.601791 25748 raft_consensus.cc:348] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.601922 25748 raft_consensus.cc:374] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:54.601950 25748 raft_consensus.cc:729] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 908cdccc69d4449aa6c9c758d47fe5b7, State: Initialized, Role: FOLLOWER
I20260925 15:01:54.602061 25748 consensus_queue.cc:260] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.602314 25748 ts_tablet_manager.cc:1428] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:01:54.602336 25459 heartbeater.cc:503] Master 127.8.13.254:44927 was elected leader, sending a full tablet report...
W20260925 15:01:54.644472 25594 tablet.cc:2369] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:54.702577 25752 raft_consensus.cc:482] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:54.702723 25752 raft_consensus.cc:504] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.703094 25752 leader_election.cc:288] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 44e31fc8cfc8410cbec4f3c2e6a87003 (127.8.13.195:43535), a8bd5d7a49274738bfa387d9d0441bd3 (127.8.13.194:43807)
I20260925 15:01:54.706049 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.703383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:58711 (local address 127.8.13.195:43535)
0925 15:01:54.703430 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.703432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.703439 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:01:54.703464 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.703465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.703474 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.703523 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:54.703704 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.704200 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705270 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.705718 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705784 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.705793 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.705939 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.705979 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:01:54.706049 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.703423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:53083 (local address 127.8.13.194:43807)
0925 15:01:54.703472 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.703474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.703589 (+ 115us) server_negotiation.cc:407] Connection header received
0925 15:01:54.703621 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.703622 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.703636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.703688 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:54.703857 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.704334 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705302 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.705702 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705774 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.705786 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.705933 (+ 147us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.705974 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:01:54.706082 25461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.703278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43535 (local address 127.8.13.193:58711)
0925 15:01:54.703342 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.703353 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.703395 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.703555 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.703556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.703558 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:54.703639 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.703641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.704274 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.704275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.705185 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.705188 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.705799 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.705878 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.705881 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.705925 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.705971 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":1}
I20260925 15:01:54.706103 25753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.703305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:43807 (local address 127.8.13.193:53083)
0925 15:01:54.703456 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.703466 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:54.703557 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:54.703700 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:54.703701 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:54.703703 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:54.703793 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.703795 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.704420 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.704421 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.705203 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:54.705206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.705797 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.705798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:54.705867 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.705871 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:54.705926 (+ 55us) client_negotiation.cc:240] Negotiation successful
0925 15:01:54.705974 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260925 15:01:54.706457 25547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ef7871a52547afb73187698affb045" candidate_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" is_pre_election: true
I20260925 15:01:54.706455 25681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ef7871a52547afb73187698affb045" candidate_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" is_pre_election: true
I20260925 15:01:54.706625 25681 raft_consensus.cc:2454] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908cdccc69d4449aa6c9c758d47fe5b7 in term 0.
I20260925 15:01:54.706625 25547 raft_consensus.cc:2454] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 908cdccc69d4449aa6c9c758d47fe5b7 in term 0.
I20260925 15:01:54.706858 25349 leader_election.cc:302] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 908cdccc69d4449aa6c9c758d47fe5b7, a8bd5d7a49274738bfa387d9d0441bd3; no voters:
I20260925 15:01:54.707005 25752 raft_consensus.cc:2781] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:54.707090 25752 raft_consensus.cc:482] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:54.707111 25752 raft_consensus.cc:3037] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:54.707921 25752 raft_consensus.cc:504] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.708069 25752 leader_election.cc:288] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [CANDIDATE]: Term 1 election: Requested vote from peers 44e31fc8cfc8410cbec4f3c2e6a87003 (127.8.13.195:43535), a8bd5d7a49274738bfa387d9d0441bd3 (127.8.13.194:43807)
I20260925 15:01:54.708246 25681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ef7871a52547afb73187698affb045" candidate_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003"
I20260925 15:01:54.708248 25547 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31ef7871a52547afb73187698affb045" candidate_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8bd5d7a49274738bfa387d9d0441bd3"
I20260925 15:01:54.708338 25547 raft_consensus.cc:3037] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:54.708336 25681 raft_consensus.cc:3037] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:54.709134 25681 raft_consensus.cc:2454] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908cdccc69d4449aa6c9c758d47fe5b7 in term 1.
I20260925 15:01:54.709219 25547 raft_consensus.cc:2454] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 908cdccc69d4449aa6c9c758d47fe5b7 in term 1.
I20260925 15:01:54.709373 25348 leader_election.cc:302] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44e31fc8cfc8410cbec4f3c2e6a87003, 908cdccc69d4449aa6c9c758d47fe5b7; no voters:
I20260925 15:01:54.709458 25752 raft_consensus.cc:2781] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:54.709590 25752 raft_consensus.cc:686] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 1 LEADER]: Becoming Leader. State: Replica: 908cdccc69d4449aa6c9c758d47fe5b7, State: Running, Role: LEADER
I20260925 15:01:54.709681 25752 consensus_queue.cc:237] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } }
I20260925 15:01:54.710567 25268 catalog_manager.cc:5495] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 908cdccc69d4449aa6c9c758d47fe5b7 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "908cdccc69d4449aa6c9c758d47fe5b7" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41475 } health_report { overall_health: HEALTHY } } }
I20260925 15:01:54.734498 25743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.732137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36252 (local address 127.8.13.193:41475)
0925 15:01:54.732204 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.732206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.732224 (+ 18us) server_negotiation.cc:407] Connection header received
0925 15:01:54.732263 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.732263 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.732284 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.732341 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:54.732472 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.732954 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.733755 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.733947 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.734003 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.734163 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:54.734274 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:54.734350 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.734368 (+ 18us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.734393 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:01:54.736385 25547 raft_consensus.cc:1261] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 1 FOLLOWER]: Refusing update from remote peer 908cdccc69d4449aa6c9c758d47fe5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:54.736405 25681 raft_consensus.cc:1261] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 1 FOLLOWER]: Refusing update from remote peer 908cdccc69d4449aa6c9c758d47fe5b7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260925 15:01:54.736608 25460 tablet.cc:2369] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:54.736744 25754 consensus_queue.cc:1035] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8bd5d7a49274738bfa387d9d0441bd3" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 43807 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:54.736867 25752 consensus_queue.cc:1035] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44e31fc8cfc8410cbec4f3c2e6a87003" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:54.739851 25758 mvcc.cc:205] Tried to move back new op lower bound from 7333267516357754880 to 7333267516251533312. Current Snapshot: MvccSnapshot[applied={T|T < 7333267516357754880}]
I20260925 15:01:54.741719 25268 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:54.741856 25268 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49972:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:54.742587 25268 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 08cfafe9a29744c5ad30c3af07ae8c79: Sending DeleteTablet for 3 replicas of tablet 31ef7871a52547afb73187698affb045
I20260925 15:01:54.743048 25393 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ef7871a52547afb73187698affb045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:54 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:36242
I20260925 15:01:54.743093 25527 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ef7871a52547afb73187698affb045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:54 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:55938
I20260925 15:01:54.743279 25767 tablet_replica.cc:330] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: stopping tablet replica
I20260925 15:01:54.743369 25767 raft_consensus.cc:2229] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:54.743336 25661 tablet_service.cc:1514] Processing DeleteTablet for tablet 31ef7871a52547afb73187698affb045 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:54 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:45180
I20260925 15:01:54.743469 25767 raft_consensus.cc:2258] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:54.743607 25769 tablet_replica.cc:330] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: stopping tablet replica
I20260925 15:01:54.743757 25769 raft_consensus.cc:2229] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:54.743933 25769 raft_consensus.cc:2258] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:54.743933 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25330
I20260925 15:01:54.743973 25767 ts_tablet_manager.cc:1905] T 31ef7871a52547afb73187698affb045 P 908cdccc69d4449aa6c9c758d47fe5b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:54.744724 25769 ts_tablet_manager.cc:1905] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:54.746059 25769 ts_tablet_manager.cc:1918] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:54.746121 25769 log.cc:1196] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/31ef7871a52547afb73187698affb045
I20260925 15:01:54.746331 25769 ts_tablet_manager.cc:1939] T 31ef7871a52547afb73187698affb045 P 44e31fc8cfc8410cbec4f3c2e6a87003: Deleting consensus metadata
I20260925 15:01:54.746703 25254 catalog_manager.cc:4841] TS 44e31fc8cfc8410cbec4f3c2e6a87003 (127.8.13.195:43535): tablet 31ef7871a52547afb73187698affb045 (table test-table [id=c921e5e6343e44b2bba0374e8bcd616a]) successfully deleted
I20260925 15:01:54.747668 25768 tablet_replica.cc:330] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: stopping tablet replica
I20260925 15:01:54.747782 25768 raft_consensus.cc:2229] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:54.747896 25768 raft_consensus.cc:2258] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:54.748268 25768 ts_tablet_manager.cc:1905] T 31ef7871a52547afb73187698affb045 P a8bd5d7a49274738bfa387d9d0441bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260925 15:01:54.749249 25254 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:41475 (error 108)
I20260925 15:01:54.749496 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25464
I20260925 15:01:54.749770 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.749655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41475 (local address 127.0.0.1:36260)
0925 15:01:54.749700 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.749730 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41475: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:01:54.750057 25254 catalog_manager.cc:4584] TS 908cdccc69d4449aa6c9c758d47fe5b7 (127.8.13.193:41475): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 31ef7871a52547afb73187698affb045: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41475: connect: Connection refused (error 111)
I20260925 15:01:54.754182 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25598
I20260925 15:01:54.754208 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.754101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:43807 (local address 127.0.0.1:55954)
0925 15:01:54.754150 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:54.754177 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.194:43807: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:01:54.758200 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25236
2026-09-25T15:01:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1917 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-25T15:01:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:54Z Disabled control of system clock
I20260925 15:01:54.784173 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40117
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:54.864284 25774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:54.864523 25774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:54.864545 25774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:54.865777 25774 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:54.865824 25774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:54.865835 25774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:54.865846 25774 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:54.865857 25774 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:54.867264 25774 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:40117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:54.867514 25774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:54.867825 25774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:54.872427 25779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.872397 25780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:54.872402 25782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:54.872439 25774 server_base.cc:1034] running on GCE node
I20260925 15:01:54.872866 25774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:54.873133 25774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:54.874289 25774 hybrid_clock.cc:648] HybridClock initialized: now 1790348514874277 us; error 28 us; skew 500 ppm
I20260925 15:01:54.875540 25774 webserver.cc:460] Webserver started at http://127.8.13.254:42011/ using document root <none> and password file <none>
I20260925 15:01:54.875793 25774 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:54.875839 25774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:54.875967 25774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:54.877027 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "45cbe197da4e4f5bba150516da248c2a"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "51fa24f9ae9c25906747b171ecfdfff3"
server_key_iv: "d5d0b033e4d8d417fac9860aab6805ca"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.877369 25774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "45cbe197da4e4f5bba150516da248c2a"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "51fa24f9ae9c25906747b171ecfdfff3"
server_key_iv: "d5d0b033e4d8d417fac9860aab6805ca"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.878621 25774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:54.879309 25788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:54.879477 25774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:54.879587 25774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "45cbe197da4e4f5bba150516da248c2a"
format_stamp: "Formatted at 2026-09-25 15:01:54 on dist-test-slave-15hz"
server_key: "51fa24f9ae9c25906747b171ecfdfff3"
server_key_iv: "d5d0b033e4d8d417fac9860aab6805ca"
server_key_version: "encryptionkey@0"
I20260925 15:01:54.879654 25774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:54.893179 25774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:54.893489 25774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:54.893596 25774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:54.897230 25774 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:40117
I20260925 15:01:54.897289 25840 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:40117 every 8 connection(s)
I20260925 15:01:54.897605 25774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:54.898531 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25774
I20260925 15:01:54.898643 25841 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:54.898643 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:54.898874 8247 external_mini_cluster.cc:1428] Setting key 7bd00ed384b60fba4d6d9b5bc6d7d5d9
I20260925 15:01:54.901718 25841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: Bootstrap starting.
I20260925 15:01:54.902448 25841 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:54.902724 25841 log.cc:824] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:54.903342 25841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: No bootstrap required, opened a new log
I20260925 15:01:54.904369 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:54.899393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47464 (local address 127.8.13.254:40117)
0925 15:01:54.899635 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:54.899643 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:54.899728 (+ 85us) server_negotiation.cc:407] Connection header received
0925 15:01:54.900192 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:54.900199 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:54.900273 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:54.900385 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:54.900604 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.901459 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.902470 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:54.902918 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:54.903031 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:54.903719 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:54.903723 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:54.903723 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:54.903896 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:54.903930 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:54.903940 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:54.904004 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:54.904087 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:01:54.904156 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260925 15:01:54.905375 25841 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } }
I20260925 15:01:54.905514 25841 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:54.905534 25841 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45cbe197da4e4f5bba150516da248c2a, State: Initialized, Role: FOLLOWER
I20260925 15:01:54.905616 25841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } }
I20260925 15:01:54.905668 25841 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:54.905689 25841 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:54.905718 25841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:54.906220 25841 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } }
I20260925 15:01:54.906307 25841 leader_election.cc:302] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45cbe197da4e4f5bba150516da248c2a; no voters:
I20260925 15:01:54.906455 25841 leader_election.cc:288] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:54.906509 25846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:54.906646 25846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [term 1 LEADER]: Becoming Leader. State: Replica: 45cbe197da4e4f5bba150516da248c2a, State: Running, Role: LEADER
I20260925 15:01:54.906684 25841 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:54.906733 25846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } }
I20260925 15:01:54.907083 25847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45cbe197da4e4f5bba150516da248c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } } }
I20260925 15:01:54.907181 25847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:54.907100 25848 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45cbe197da4e4f5bba150516da248c2a. Latest consensus state: current_term: 1 leader_uuid: "45cbe197da4e4f5bba150516da248c2a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45cbe197da4e4f5bba150516da248c2a" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 40117 } } }
I20260925 15:01:54.907306 25848 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:54.907718 25855 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:54.908178 25855 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:54.909701 25855 catalog_manager.cc:1347] Generated new cluster ID: 494e517772e9431cbd0aa1fc35d3ebad
I20260925 15:01:54.909757 25855 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:54.928054 25855 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:54.928592 25855 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:54.940315 25855 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: Generated new TSK 0
I20260925 15:01:54.940507 25855 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:54.943826 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--builtin_ntp_servers=127.8.13.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:55.024382 25865 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:55.024590 25865 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:55.024611 25865 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:55.025830 25865 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:55.025878 25865 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:55.025904 25865 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:55.027298 25865 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25865
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:55.027583 25865 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:55.027899 25865 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:55.033880 25865 server_base.cc:1034] running on GCE node
W20260925 15:01:55.033849 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.033845 25870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.033849 25871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.034339 25865 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:55.034614 25865 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:55.035764 25865 hybrid_clock.cc:648] HybridClock initialized: now 1790348515035749 us; error 35 us; skew 500 ppm
I20260925 15:01:55.037122 25865 webserver.cc:460] Webserver started at http://127.8.13.193:34529/ using document root <none> and password file <none>
I20260925 15:01:55.037353 25865 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:55.037405 25865 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:55.037533 25865 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:55.038627 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "167ccd7fac85473a93dd454673015cc6"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "574b3c652eb88e4a60a996ac8e5ac77b"
server_key_iv: "f4bcef5bcbe7eb584c832a63877aa2f1"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.038975 25865 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "167ccd7fac85473a93dd454673015cc6"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "574b3c652eb88e4a60a996ac8e5ac77b"
server_key_iv: "f4bcef5bcbe7eb584c832a63877aa2f1"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.040354 25865 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:55.041136 25879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.041293 25865 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:01:55.041376 25865 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "167ccd7fac85473a93dd454673015cc6"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "574b3c652eb88e4a60a996ac8e5ac77b"
server_key_iv: "f4bcef5bcbe7eb584c832a63877aa2f1"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.041440 25865 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:55.063860 25865 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:55.064164 25865 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:55.064307 25865 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:55.064522 25865 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:55.064826 25865 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:55.064857 25865 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.064885 25865 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:55.064910 25865 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.071084 25865 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:38817
I20260925 15:01:55.071133 25992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:38817 every 8 connection(s)
I20260925 15:01:55.071488 25865 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:55.076536 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.072276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:40451 (local address 127.8.13.254:40117)
0925 15:01:55.072348 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.072350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.072557 (+ 207us) server_negotiation.cc:407] Connection header received
0925 15:01:55.073051 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.073052 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.073066 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.073119 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.073794 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.074291 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.075251 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.075432 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.075478 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.076316 (+ 838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.076317 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.076318 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:55.076323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.076337 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:55.076341 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.076381 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.076438 (+ 57us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.076468 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260925 15:01:55.076762 25995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.072194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40117 (local address 127.8.13.193:40451)
0925 15:01:55.072437 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.072474 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.072971 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.073191 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.073194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.073202 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:55.073700 (+ 498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.073704 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.074360 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.074360 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.075181 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.075184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.075499 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.075499 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.075671 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.076173 (+ 502us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:55.076174 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:55.076204 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:55.076235 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:55.076386 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:55.076389 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:55.076392 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.076439 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.076529 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":83,"threads_started":1}
I20260925 15:01:55.077306 25993 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40117
I20260925 15:01:55.077417 25993 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:55.077611 25993 heartbeater.cc:511] Master 127.8.13.254:40117 requested a full tablet report, sending...
I20260925 15:01:55.078109 25805 ts_manager.cc:194] Registered new tserver with Master: 167ccd7fac85473a93dd454673015cc6 (127.8.13.193:38817)
I20260925 15:01:55.078720 25805 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.193:40451
I20260925 15:01:55.078809 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25865
I20260925 15:01:55.078908 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:55.079120 8247 external_mini_cluster.cc:1428] Setting key 7d61164f0492a4604a83bc86a470ed51
I20260925 15:01:55.079960 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--builtin_ntp_servers=127.8.13.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:55.159076 25996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:55.159308 25996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:55.159338 25996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:55.160521 25996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:55.160567 25996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:55.160601 25996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:55.161978 25996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:55.162233 25996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:55.162534 25996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:55.167402 26001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.167562 25996 server_base.cc:1034] running on GCE node
W20260925 15:01:55.167375 26002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.167382 26004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.167905 25996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:55.168123 25996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:55.169257 25996 hybrid_clock.cc:648] HybridClock initialized: now 1790348515169248 us; error 26 us; skew 500 ppm
I20260925 15:01:55.170439 25996 webserver.cc:460] Webserver started at http://127.8.13.194:36977/ using document root <none> and password file <none>
I20260925 15:01:55.170684 25996 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:55.170733 25996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:55.170864 25996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:55.172026 25996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "6d9a6601b8184e0e9ffd7db729291f8f"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "7eb7a333d39f5eed00178820d0963138"
server_key_iv: "7fecd83d02638eeb6f45f4c1bba95702"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.172375 25996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "6d9a6601b8184e0e9ffd7db729291f8f"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "7eb7a333d39f5eed00178820d0963138"
server_key_iv: "7fecd83d02638eeb6f45f4c1bba95702"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.173686 25996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:55.174408 26010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.174548 25996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:55.174628 25996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "6d9a6601b8184e0e9ffd7db729291f8f"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "7eb7a333d39f5eed00178820d0963138"
server_key_iv: "7fecd83d02638eeb6f45f4c1bba95702"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.174705 25996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:55.190275 25996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:55.190586 25996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:55.190722 25996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:55.190939 25996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:55.191272 25996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:55.191305 25996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.191334 25996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:55.191361 25996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.196698 25996 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:46009
I20260925 15:01:55.196763 26123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:46009 every 8 connection(s)
I20260925 15:01:55.197060 25996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:55.202067 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.197756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:48211 (local address 127.8.13.254:40117)
0925 15:01:55.197814 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.197816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.198018 (+ 202us) server_negotiation.cc:407] Connection header received
0925 15:01:55.198591 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.198593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.198605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.198652 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.199321 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.199840 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.200784 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.200983 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.201038 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.201808 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.201809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.201809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:55.201814 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.201829 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:55.201833 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.201878 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.201960 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.201985 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:01:55.202271 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.197733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40117 (local address 127.8.13.194:48211)
0925 15:01:55.197908 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.197941 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.198491 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.198735 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.198740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.198748 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:55.199226 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.199230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.199923 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.199924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.200697 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.200698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.201054 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.201055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.201175 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.201674 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:55.201674 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:55.201703 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:55.201726 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:55.201888 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:55.201893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:55.201896 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.201955 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.202055 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":76,"threads_started":1}
I20260925 15:01:55.202728 26124 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40117
I20260925 15:01:55.202807 26124 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:55.202984 26124 heartbeater.cc:511] Master 127.8.13.254:40117 requested a full tablet report, sending...
I20260925 15:01:55.203327 25805 ts_manager.cc:194] Registered new tserver with Master: 6d9a6601b8184e0e9ffd7db729291f8f (127.8.13.194:46009)
I20260925 15:01:55.203714 25805 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.194:48211
I20260925 15:01:55.204574 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 25996
I20260925 15:01:55.204661 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:55.204811 8247 external_mini_cluster.cc:1428] Setting key 549d8919f9b574c72a3da20afabc1b12
I20260925 15:01:55.205629 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--builtin_ntp_servers=127.8.13.212:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:55.285597 26127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:55.285802 26127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:55.285823 26127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:55.286934 26127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:55.286969 26127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:55.286993 26127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:55.288359 26127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:55.288564 26127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:55.288816 26127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:55.293869 26127 server_base.cc:1034] running on GCE node
W20260925 15:01:55.293854 26133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.293838 26132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:55.293851 26135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:55.294282 26127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:55.294533 26127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:55.295667 26127 hybrid_clock.cc:648] HybridClock initialized: now 1790348515295654 us; error 28 us; skew 500 ppm
I20260925 15:01:55.296880 26127 webserver.cc:460] Webserver started at http://127.8.13.195:38417/ using document root <none> and password file <none>
I20260925 15:01:55.297111 26127 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:55.297183 26127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:55.297308 26127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:55.298374 26127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "81e524727ab449898743bc11cd2a2975"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "fe093a161ecef1b769e5c3f73d9b478e"
server_key_iv: "e66e92b8b43674aec6f7b63761131be0"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.298756 26127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "81e524727ab449898743bc11cd2a2975"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "fe093a161ecef1b769e5c3f73d9b478e"
server_key_iv: "e66e92b8b43674aec6f7b63761131be0"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.300017 26127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:01:55.300740 26141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.300907 26127 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:55.300988 26127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "81e524727ab449898743bc11cd2a2975"
format_stamp: "Formatted at 2026-09-25 15:01:55 on dist-test-slave-15hz"
server_key: "fe093a161ecef1b769e5c3f73d9b478e"
server_key_iv: "e66e92b8b43674aec6f7b63761131be0"
server_key_version: "encryptionkey@0"
I20260925 15:01:55.301051 26127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:55.309834 26127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:55.310180 26127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:55.310320 26127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:55.310541 26127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:55.310853 26127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:55.310885 26127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.310913 26127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:55.310940 26127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:55.316464 26127 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:40733
I20260925 15:01:55.316524 26254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:40733 every 8 connection(s)
I20260925 15:01:55.316862 26127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:55.320258 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26127
I20260925 15:01:55.320374 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:55.320578 8247 external_mini_cluster.cc:1428] Setting key d423103c34e4db9d43cfe9dd17b16da4
I20260925 15:01:55.322300 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.317574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:41937 (local address 127.8.13.254:40117)
0925 15:01:55.317638 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.317641 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.317865 (+ 224us) server_negotiation.cc:407] Connection header received
0925 15:01:55.318387 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.318388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.318401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.318454 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.319147 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.319677 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.320814 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.321079 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.321117 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.322044 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.322045 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.322046 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:55.322051 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.322065 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:55.322070 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.322115 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.322190 (+ 75us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.322218 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:01:55.322508 26257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.317532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:40117 (local address 127.8.13.195:41937)
0925 15:01:55.317742 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.317778 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.318287 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.318526 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.318531 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.318541 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:55.319049 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.319054 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.319763 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.319764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.320704 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.320707 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.321211 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.321212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.321365 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.321890 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:55.321891 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:55.321945 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:55.321970 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:55.322122 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:55.322127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:55.322131 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.322194 (+ 63us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.322303 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":90,"threads_started":1}
I20260925 15:01:55.322947 26255 heartbeater.cc:347] Connected to a master server at 127.8.13.254:40117
I20260925 15:01:55.323035 26255 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:55.323246 26255 heartbeater.cc:511] Master 127.8.13.254:40117 requested a full tablet report, sending...
I20260925 15:01:55.323649 25805 ts_manager.cc:194] Registered new tserver with Master: 81e524727ab449898743bc11cd2a2975 (127.8.13.195:40733)
I20260925 15:01:55.324189 25805 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.195:41937
I20260925 15:01:55.332553 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:55.336669 25844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.333929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47468 (local address 127.8.13.254:40117)
0925 15:01:55.334010 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.334012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.334077 (+ 65us) server_negotiation.cc:407] Connection header received
0925 15:01:55.334146 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.334146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.334157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.334206 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:55.334352 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.334848 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.335625 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.335826 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.335880 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.336436 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:55.336438 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:55.336438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:55.336442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:55.336456 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:55.336460 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:55.336505 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.336554 (+ 49us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.336577 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260925 15:01:55.337906 25805 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47468:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:55.339401 25805 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:55.345857 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60086 (local address 127.8.13.195:40733)
0925 15:01:55.341822 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.341826 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.341837 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:01:55.341878 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.341880 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.341895 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.341978 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:55.342287 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.343456 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.344772 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.345497 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.345601 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.345704 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.345739 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.345776 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260925 15:01:55.345847 26264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:46009 (local address 127.0.0.1:50200)
0925 15:01:55.341636 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.341646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.341699 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.342158 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.342159 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.342161 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:55.342210 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.342211 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.343191 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.343192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.344499 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.344502 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.345485 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.345486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.345583 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.345600 (+ 17us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.345682 (+ 82us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.345746 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":82,"threads_started":1}
I20260925 15:01:55.345858 26263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40733 (local address 127.0.0.1:60086)
0925 15:01:55.341485 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.341526 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.341587 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.341994 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.341997 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.341999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:55.342076 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.342079 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.343582 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.343583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.344664 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.344666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.345610 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.345611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.345676 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.345679 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.345714 (+ 35us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.345766 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":86,"threads_started":1}
I20260925 15:01:55.346122 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50200 (local address 127.8.13.194:46009)
0925 15:01:55.341996 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.342000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.342010 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:55.342044 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.342045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.342061 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.342141 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:55.342401 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.343109 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.344596 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.345373 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.345448 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.345541 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.345991 (+ 450us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.346045 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":86,"threads_started":1}
I20260925 15:01:55.346971 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:38817 (local address 127.0.0.1:56896)
0925 15:01:55.341784 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.341792 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.341825 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.342254 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.342255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.342258 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:55.342306 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.342308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.344754 (+ 2446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.344755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.345777 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.345779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.346734 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.346736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.346800 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.346805 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.346857 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.346900 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":79,"threads_started":1}
I20260925 15:01:55.346944 26189 tablet_service.cc:1467] Processing CreateTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d (DEFAULT_TABLE table=test-table [id=f0e7696b0d434776886419d6be48dcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:55.346941 26058 tablet_service.cc:1467] Processing CreateTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d (DEFAULT_TABLE table=test-table [id=f0e7696b0d434776886419d6be48dcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:55.347254 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.341633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56896 (local address 127.8.13.193:38817)
0925 15:01:55.342031 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.342036 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.342045 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:01:55.342081 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.342083 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.342097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.342195 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:55.342375 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.343456 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.345867 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.346645 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.346714 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.346797 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.347115 (+ 318us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.347174 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":93,"threads_started":1}
I20260925 15:01:55.347298 26058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c3cd48509e484ebdfb1e7b80be808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:55.347307 26189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c3cd48509e484ebdfb1e7b80be808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:55.348196 25927 tablet_service.cc:1467] Processing CreateTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d (DEFAULT_TABLE table=test-table [id=f0e7696b0d434776886419d6be48dcec]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:55.348562 25927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50c3cd48509e484ebdfb1e7b80be808d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:55.350018 26269 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Bootstrap starting.
I20260925 15:01:55.350617 26269 tablet_bootstrap.cc:654] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:55.350900 26269 log.cc:824] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:55.351565 26269 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: No bootstrap required, opened a new log
I20260925 15:01:55.351671 26269 ts_tablet_manager.cc:1397] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:55.351933 26272 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: Bootstrap starting.
I20260925 15:01:55.352044 26270 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Bootstrap starting.
I20260925 15:01:55.352567 26270 tablet_bootstrap.cc:654] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:55.352649 26272 tablet_bootstrap.cc:654] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:55.352840 26270 log.cc:824] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:55.352952 26272 log.cc:824] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:55.353549 26270 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: No bootstrap required, opened a new log
I20260925 15:01:55.353610 26272 tablet_bootstrap.cc:492] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: No bootstrap required, opened a new log
I20260925 15:01:55.353627 26270 ts_tablet_manager.cc:1397] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:55.353703 26272 ts_tablet_manager.cc:1397] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:55.354293 26269 raft_consensus.cc:348] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.354477 26269 raft_consensus.cc:374] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:55.354508 26269 raft_consensus.cc:729] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81e524727ab449898743bc11cd2a2975, State: Initialized, Role: FOLLOWER
I20260925 15:01:55.354632 26269 consensus_queue.cc:260] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.354889 26269 ts_tablet_manager.cc:1428] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:55.354919 26255 heartbeater.cc:503] Master 127.8.13.254:40117 was elected leader, sending a full tablet report...
I20260925 15:01:55.355866 26272 raft_consensus.cc:348] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.356027 26272 raft_consensus.cc:374] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:55.356062 26272 raft_consensus.cc:729] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 167ccd7fac85473a93dd454673015cc6, State: Initialized, Role: FOLLOWER
I20260925 15:01:55.356086 26270 raft_consensus.cc:348] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.356173 26272 consensus_queue.cc:260] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.356247 26270 raft_consensus.cc:374] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:55.356273 26270 raft_consensus.cc:729] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d9a6601b8184e0e9ffd7db729291f8f, State: Initialized, Role: FOLLOWER
I20260925 15:01:55.356443 26272 ts_tablet_manager.cc:1428] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:55.356545 25993 heartbeater.cc:503] Master 127.8.13.254:40117 was elected leader, sending a full tablet report...
I20260925 15:01:55.356815 26270 consensus_queue.cc:260] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.357149 26270 ts_tablet_manager.cc:1428] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:01:55.357265 26124 heartbeater.cc:503] Master 127.8.13.254:40117 was elected leader, sending a full tablet report...
I20260925 15:01:55.380872 26277 raft_consensus.cc:482] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:55.380990 26277 raft_consensus.cc:504] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.381343 26277 leader_election.cc:288] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 81e524727ab449898743bc11cd2a2975 (127.8.13.195:40733), 167ccd7fac85473a93dd454673015cc6 (127.8.13.193:38817)
I20260925 15:01:55.385324 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.381562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:52035 (local address 127.8.13.193:38817)
0925 15:01:55.381604 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.381606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.381674 (+ 68us) server_negotiation.cc:407] Connection header received
0925 15:01:55.381731 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.381732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.381748 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.381801 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:55.381957 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.382767 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.384347 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.384943 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.385032 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.385040 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.385201 (+ 161us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.385249 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:01:55.385324 26266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.381563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:35105 (local address 127.8.13.195:40733)
0925 15:01:55.381609 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.381611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.381794 (+ 183us) server_negotiation.cc:407] Connection header received
0925 15:01:55.381833 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.381834 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.381844 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.381892 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:55.382014 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.382754 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.384347 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.384943 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.385031 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.385040 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.385202 (+ 162us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.385251 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:01:55.385332 26126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.381494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:38817 (local address 127.8.13.194:52035)
0925 15:01:55.381613 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.381624 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.381677 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.381814 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.381815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.381817 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:55.381883 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.381885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.382844 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.382845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.384230 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.384246 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.385042 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.385043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.385127 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.385131 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.385183 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.385232 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":53}
I20260925 15:01:55.385337 26278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.381495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:40733 (local address 127.8.13.194:35105)
0925 15:01:55.381730 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.381739 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:55.381788 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:55.381902 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:55.381903 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:55.381905 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:55.381958 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.381959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.382844 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.382845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.384230 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:55.384234 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.385042 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.385043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:55.385127 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.385132 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:01:55.385183 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:01:55.385229 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"mutex_wait_us":115,"thread_start_us":110,"threads_started":1}
I20260925 15:01:55.385694 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c3cd48509e484ebdfb1e7b80be808d" candidate_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e524727ab449898743bc11cd2a2975" is_pre_election: true
I20260925 15:01:55.385864 26209 raft_consensus.cc:2454] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d9a6601b8184e0e9ffd7db729291f8f in term 0.
I20260925 15:01:55.385824 25947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c3cd48509e484ebdfb1e7b80be808d" candidate_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167ccd7fac85473a93dd454673015cc6" is_pre_election: true
I20260925 15:01:55.385972 25947 raft_consensus.cc:2454] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6d9a6601b8184e0e9ffd7db729291f8f in term 0.
I20260925 15:01:55.386116 26012 leader_election.cc:302] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d9a6601b8184e0e9ffd7db729291f8f, 81e524727ab449898743bc11cd2a2975; no voters:
I20260925 15:01:55.386267 26277 raft_consensus.cc:2781] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:55.386336 26277 raft_consensus.cc:482] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:55.386360 26277 raft_consensus.cc:3037] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:55.387053 26277 raft_consensus.cc:504] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.387225 26277 leader_election.cc:288] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [CANDIDATE]: Term 1 election: Requested vote from peers 81e524727ab449898743bc11cd2a2975 (127.8.13.195:40733), 167ccd7fac85473a93dd454673015cc6 (127.8.13.193:38817)
I20260925 15:01:55.387378 25947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c3cd48509e484ebdfb1e7b80be808d" candidate_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "167ccd7fac85473a93dd454673015cc6"
I20260925 15:01:55.387378 26209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50c3cd48509e484ebdfb1e7b80be808d" candidate_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "81e524727ab449898743bc11cd2a2975"
I20260925 15:01:55.387456 26209 raft_consensus.cc:3037] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:55.387456 25947 raft_consensus.cc:3037] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:55.388382 26209 raft_consensus.cc:2454] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9a6601b8184e0e9ffd7db729291f8f in term 1.
I20260925 15:01:55.388382 25947 raft_consensus.cc:2454] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6d9a6601b8184e0e9ffd7db729291f8f in term 1.
I20260925 15:01:55.388562 26011 leader_election.cc:302] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 167ccd7fac85473a93dd454673015cc6, 6d9a6601b8184e0e9ffd7db729291f8f; no voters:
I20260925 15:01:55.388669 26277 raft_consensus.cc:2781] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:55.388825 26277 raft_consensus.cc:686] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 1 LEADER]: Becoming Leader. State: Replica: 6d9a6601b8184e0e9ffd7db729291f8f, State: Running, Role: LEADER
I20260925 15:01:55.388921 26277 consensus_queue.cc:237] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } }
I20260925 15:01:55.389765 25804 catalog_manager.cc:5495] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f reported cstate change: term changed from 0 to 1, leader changed from <none> to 6d9a6601b8184e0e9ffd7db729291f8f (127.8.13.194). New cstate: current_term: 1 leader_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d9a6601b8184e0e9ffd7db729291f8f" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:55.405855 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.403686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.8.13.194:46009)
0925 15:01:55.403759 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:55.403761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:55.403782 (+ 21us) server_negotiation.cc:407] Connection header received
0925 15:01:55.403819 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:55.403820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:55.403838 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:55.403897 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:55.403992 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.404453 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.405215 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:55.405406 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:55.405456 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:55.405553 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:55.405653 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:55.405706 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:55.405719 (+ 13us) server_negotiation.cc:299] Negotiation successful
0925 15:01:55.405745 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260925 15:01:55.407752 25947 raft_consensus.cc:1261] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 1 FOLLOWER]: Refusing update from remote peer 6d9a6601b8184e0e9ffd7db729291f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:55.407752 26209 raft_consensus.cc:1261] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 1 FOLLOWER]: Refusing update from remote peer 6d9a6601b8184e0e9ffd7db729291f8f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:55.408124 26277 consensus_queue.cc:1035] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "167ccd7fac85473a93dd454673015cc6" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 38817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:55.408264 26279 consensus_queue.cc:1035] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [LEADER]: Connected to new peer: Peer: permanent_uuid: "81e524727ab449898743bc11cd2a2975" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 40733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:55.411005 26282 mvcc.cc:205] Tried to move back new op lower bound from 7333267519107325952 to 7333267519033827328. Current Snapshot: MvccSnapshot[applied={T|T < 7333267519107325952 or (T in {7333267519107325952})}]
I20260925 15:01:55.411206 26283 mvcc.cc:205] Tried to move back new op lower bound from 7333267519107325952 to 7333267519033827328. Current Snapshot: MvccSnapshot[applied={T|T < 7333267519107325952}]
I20260925 15:01:55.411295 26284 mvcc.cc:205] Tried to move back new op lower bound from 7333267519107325952 to 7333267519033827328. Current Snapshot: MvccSnapshot[applied={T|T < 7333267519107325952}]
I20260925 15:01:55.413487 25804 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:55.413583 25804 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47468:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:55.414368 25804 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 45cbe197da4e4f5bba150516da248c2a: Sending DeleteTablet for 3 replicas of tablet 50c3cd48509e484ebdfb1e7b80be808d
I20260925 15:01:55.414723 26058 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:55 UTC) from {username='slave'} at 127.0.0.1:50200
I20260925 15:01:55.414752 25927 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:55 UTC) from {username='slave'} at 127.0.0.1:56896
I20260925 15:01:55.414793 26189 tablet_service.cc:1514] Processing DeleteTablet for tablet 50c3cd48509e484ebdfb1e7b80be808d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:55 UTC) from {username='slave'} at 127.0.0.1:60086
I20260925 15:01:55.414974 26291 tablet_replica.cc:330] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6: stopping tablet replica
I20260925 15:01:55.415047 26292 tablet_replica.cc:330] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: stopping tablet replica
I20260925 15:01:55.415122 26291 raft_consensus.cc:2229] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:55.415201 26293 tablet_replica.cc:330] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: stopping tablet replica
I20260925 15:01:55.415299 26291 raft_consensus.cc:2258] T 50c3cd48509e484ebdfb1e7b80be808d P 167ccd7fac85473a93dd454673015cc6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:55.415350 26293 raft_consensus.cc:2229] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:55.415351 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25865
I20260925 15:01:55.417097 26292 raft_consensus.cc:2229] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:55.417274 26292 raft_consensus.cc:2258] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:55.417447 26293 raft_consensus.cc:2258] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:55.419209 26292 ts_tablet_manager.cc:1905] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:55.419216 26293 ts_tablet_manager.cc:1905] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:55.420822 26292 ts_tablet_manager.cc:1918] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:55.420862 26293 ts_tablet_manager.cc:1918] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:55.420957 26293 log.cc:1196] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/50c3cd48509e484ebdfb1e7b80be808d
I20260925 15:01:55.420979 26292 log.cc:1196] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/50c3cd48509e484ebdfb1e7b80be808d
I20260925 15:01:55.421222 26293 ts_tablet_manager.cc:1939] T 50c3cd48509e484ebdfb1e7b80be808d P 81e524727ab449898743bc11cd2a2975: Deleting consensus metadata
I20260925 15:01:55.421365 26292 ts_tablet_manager.cc:1939] T 50c3cd48509e484ebdfb1e7b80be808d P 6d9a6601b8184e0e9ffd7db729291f8f: Deleting consensus metadata
I20260925 15:01:55.421702 25790 catalog_manager.cc:4841] TS 81e524727ab449898743bc11cd2a2975 (127.8.13.195:40733): tablet 50c3cd48509e484ebdfb1e7b80be808d (table test-table [id=f0e7696b0d434776886419d6be48dcec]) successfully deleted
I20260925 15:01:55.422139 25792 catalog_manager.cc:4841] TS 6d9a6601b8184e0e9ffd7db729291f8f (127.8.13.194:46009): tablet 50c3cd48509e484ebdfb1e7b80be808d (table test-table [id=f0e7696b0d434776886419d6be48dcec]) successfully deleted
W20260925 15:01:55.422520 25789 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:38817 (error 108)
I20260925 15:01:55.422691 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25996
I20260925 15:01:55.423502 26265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:55.423376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:38817 (local address 127.0.0.1:56902)
0925 15:01:55.423431 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:55.423473 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:38817: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260925 15:01:55.423625 25789 catalog_manager.cc:4584] TS 167ccd7fac85473a93dd454673015cc6 (127.8.13.193:38817): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50c3cd48509e484ebdfb1e7b80be808d: Network error: Client connection negotiation failed: client connection to 127.8.13.193:38817: connect: Connection refused (error 111)
I20260925 15:01:55.428164 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26127
I20260925 15:01:55.432229 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 25774
2026-09-25T15:01:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (673 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:55 dist-test-slave-15hz krb5kdc[26297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:55 dist-test-slave-15hz krb5kdc[26297](info): set up 2 sockets
Sep 25 15:01:55 dist-test-slave-15hz krb5kdc[26297](info): commencing operation
krb5kdc: starting...
W20260925 15:01:56.379094 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:01:56.491936 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39845
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:39845
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:56.571085 26310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:56.571326 26310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:56.571362 26310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:56.572623 26310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:56.572674 26310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:56.572693 26310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:56.572710 26310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:56.572724 26310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:56.574262 26310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:39845
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:39845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:56.574523 26310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:56.574733 26310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:56.579684 26318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.579697 26315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.579722 26316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:56.579779 26310 server_base.cc:1034] running on GCE node
I20260925 15:01:56.580194 26310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:56.580497 26310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:56.581653 26310 hybrid_clock.cc:648] HybridClock initialized: now 1790348516581638 us; error 31 us; skew 500 ppm
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:56.584072 26310 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:01:56.584502 26310 webserver.cc:460] Webserver started at http://127.8.13.254:45955/ using document root <none> and password file <none>
I20260925 15:01:56.584690 26310 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:56.584728 26310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:56.584822 26310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:56.585837 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "a319b3acd53e4902b1c7664ab96d4561"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "f749845be2047168906f50f44ce47f40"
server_key_iv: "6e090962f51a322019046ff37e397f5a"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.586164 26310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "a319b3acd53e4902b1c7664ab96d4561"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "f749845be2047168906f50f44ce47f40"
server_key_iv: "6e090962f51a322019046ff37e397f5a"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.587337 26310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:56.588097 26325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.588245 26310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.588328 26310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "a319b3acd53e4902b1c7664ab96d4561"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "f749845be2047168906f50f44ce47f40"
server_key_iv: "6e090962f51a322019046ff37e397f5a"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.588392 26310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:56.608582 26310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:56.610946 26310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:56.611085 26310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:56.615818 26310 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:39845
I20260925 15:01:56.615861 26377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:39845 every 8 connection(s)
I20260925 15:01:56.616245 26310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:56.616585 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26310
I20260925 15:01:56.616684 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:56.616858 8247 external_mini_cluster.cc:1428] Setting key dd63ae71c82e5b42ba457ade66ce556a
I20260925 15:01:56.617347 26378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:56.619944 26378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: Bootstrap starting.
I20260925 15:01:56.620486 26378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:56.620730 26378 log.cc:824] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:56.621315 26378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: No bootstrap required, opened a new log
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:56.623083 26378 raft_consensus.cc:348] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } }
I20260925 15:01:56.623217 26378 raft_consensus.cc:374] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:56.623250 26378 raft_consensus.cc:729] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a319b3acd53e4902b1c7664ab96d4561, State: Initialized, Role: FOLLOWER
I20260925 15:01:56.623373 26378 consensus_queue.cc:260] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } }
I20260925 15:01:56.623441 26378 raft_consensus.cc:388] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:56.623471 26378 raft_consensus.cc:482] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:56.623517 26378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:56.624208 26378 raft_consensus.cc:504] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } }
I20260925 15:01:56.624318 26378 leader_election.cc:302] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a319b3acd53e4902b1c7664ab96d4561; no voters:
I20260925 15:01:56.624508 26378 leader_election.cc:288] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:56.624552 26383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:56.624703 26383 raft_consensus.cc:686] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [term 1 LEADER]: Becoming Leader. State: Replica: a319b3acd53e4902b1c7664ab96d4561, State: Running, Role: LEADER
I20260925 15:01:56.624821 26378 sys_catalog.cc:564] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:56.624835 26383 consensus_queue.cc:237] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } }
I20260925 15:01:56.625247 26384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a319b3acd53e4902b1c7664ab96d4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } } }
I20260925 15:01:56.625319 26384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:56.625556 26385 sys_catalog.cc:455] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a319b3acd53e4902b1c7664ab96d4561. Latest consensus state: current_term: 1 leader_uuid: "a319b3acd53e4902b1c7664ab96d4561" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a319b3acd53e4902b1c7664ab96d4561" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 39845 } } }
I20260925 15:01:56.625855 26385 sys_catalog.cc:458] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:56.626015 26389 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:56.626570 26389 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:56.626785 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.617310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55856 (local address 127.8.13.254:39845)
0925 15:01:56.617498 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.617515 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.617544 (+ 29us) server_negotiation.cc:407] Connection header received
0925 15:01:56.617972 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.617978 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.618024 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.618153 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.618733 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.619674 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.620443 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.620923 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.621011 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.623126 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.623131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.623131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.623151 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.624996 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.625390 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.625393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.625394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.625499 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.625759 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.625760 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.625761 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.626070 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.626203 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.626433 (+ 230us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.626582 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260925 15:01:56.628047 26389 catalog_manager.cc:1347] Generated new cluster ID: 0ecc7ece75af4e03a04e425895eb10e1
I20260925 15:01:56.628095 26389 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:56.649791 26389 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:56.650336 26389 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:56.654944 26389 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: Generated new TSK 0
I20260925 15:01:56.655161 26389 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:01:56.682928 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:56.763959 26404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:56.764186 26404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:56.764211 26404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:56.765431 26404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:56.765477 26404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:56.765506 26404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:56.767004 26404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:56.767246 26404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:56.767467 26404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:56.772255 26412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.772312 26410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.772312 26409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:56.772379 26404 server_base.cc:1034] running on GCE node
I20260925 15:01:56.772691 26404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:56.772886 26404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:56.774053 26404 hybrid_clock.cc:648] HybridClock initialized: now 1790348516774037 us; error 26 us; skew 500 ppm
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:56.776584 26404 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:01:56.776978 26404 webserver.cc:460] Webserver started at http://127.8.13.193:42009/ using document root <none> and password file <none>
I20260925 15:01:56.777204 26404 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:56.777252 26404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:56.777379 26404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:56.778427 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "2c2e6c0dff2907db1e51760dc1f200f6"
server_key_iv: "789008fd58828567378b06d703241c21"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.778759 26404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "2c2e6c0dff2907db1e51760dc1f200f6"
server_key_iv: "789008fd58828567378b06d703241c21"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.780081 26404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:56.780793 26419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.780948 26404 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:56.781031 26404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "2c2e6c0dff2907db1e51760dc1f200f6"
server_key_iv: "789008fd58828567378b06d703241c21"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.781092 26404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:56.794859 26404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:56.797226 26404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:56.797376 26404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:56.797595 26404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:56.797901 26404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:56.797935 26404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.797966 26404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:56.797983 26404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.803337 26404 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:43107
I20260925 15:01:56.803407 26532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:43107 every 8 connection(s)
I20260925 15:01:56.803722 26404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:56.808053 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26404
I20260925 15:01:56.808151 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:56.808336 8247 external_mini_cluster.cc:1428] Setting key 06044627d5032df1347b5c27ebd82adc
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:56.812585 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.804434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:39879 (local address 127.8.13.254:39845)
0925 15:01:56.804489 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.804491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.804690 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:01:56.805161 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.805162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.805171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.805244 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.806131 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.806634 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.807604 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.807788 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.807838 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.809999 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.810000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.810001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.810009 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.811250 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.811662 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.811663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.811663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.811717 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.811960 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.811960 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.811961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.812098 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.812158 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.812408 (+ 250us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.812495 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:01:56.812765 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.804396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39845 (local address 127.8.13.193:39879)
0925 15:01:56.804551 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.804593 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.805083 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.805300 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.805303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.805864 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:56.806057 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.806061 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.806699 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.806700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.807485 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.807487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.807848 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.807849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.807959 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.808518 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:56.808528 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:56.809909 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:56.811324 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.811331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.811334 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.811590 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.811771 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.811772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.811772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.811901 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.812148 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:56.812155 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:56.812293 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.812380 (+ 87us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.812537 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
I20260925 15:01:56.813274 26533 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39845
I20260925 15:01:56.813383 26533 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:56.813585 26533 heartbeater.cc:511] Master 127.8.13.254:39845 requested a full tablet report, sending...
I20260925 15:01:56.814109 26342 ts_manager.cc:194] Registered new tserver with Master: 0e0b69f0bca246a2bb19d4efc4a4ab7d (127.8.13.193:43107)
I20260925 15:01:56.814734 26342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:39879
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:01:56.831408 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:56.910141 26538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:56.910353 26538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:56.910377 26538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:56.911578 26538 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:56.911623 26538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:56.911660 26538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:56.913159 26538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26538
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:56.913406 26538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:56.913676 26538 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:56.918728 26538 server_base.cc:1034] running on GCE node
W20260925 15:01:56.918679 26544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.918695 26546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:56.918738 26543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:56.919080 26538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:56.919318 26538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:56.920467 26538 hybrid_clock.cc:648] HybridClock initialized: now 1790348516920451 us; error 28 us; skew 500 ppm
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:56.922819 26538 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:01:56.923238 26538 webserver.cc:460] Webserver started at http://127.8.13.194:38037/ using document root <none> and password file <none>
I20260925 15:01:56.923449 26538 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:56.923496 26538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:56.923645 26538 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:56.924732 26538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "075dc09318b249a1b7e50092c576a75a"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "a8110173f0d7625b49ecf74672fd0ce0"
server_key_iv: "163d665cc6334170aa697e76bc33ebb3"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.925094 26538 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "075dc09318b249a1b7e50092c576a75a"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "a8110173f0d7625b49ecf74672fd0ce0"
server_key_iv: "163d665cc6334170aa697e76bc33ebb3"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.926595 26538 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:56.927304 26553 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.927459 26538 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:56.927559 26538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "075dc09318b249a1b7e50092c576a75a"
format_stamp: "Formatted at 2026-09-25 15:01:56 on dist-test-slave-15hz"
server_key: "a8110173f0d7625b49ecf74672fd0ce0"
server_key_iv: "163d665cc6334170aa697e76bc33ebb3"
server_key_version: "encryptionkey@0"
I20260925 15:01:56.927654 26538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:56.940250 26538 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:56.941968 26538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:56.942075 26538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:56.942268 26538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:56.942507 26538 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:56.942535 26538 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.942555 26538 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:56.942572 26538 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:56.948041 26538 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:34961
I20260925 15:01:56.948083 26666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:34961 every 8 connection(s)
I20260925 15:01:56.948370 26538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
Sep 25 15:01:56 dist-test-slave-15hz krb5kdc[26297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348516, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:56.955965 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26538
I20260925 15:01:56.956070 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:56.956264 8247 external_mini_cluster.cc:1428] Setting key 823b2b59dafd487163c6dd6c58d726ca
I20260925 15:01:56.957330 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.949099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:40439 (local address 127.8.13.254:39845)
0925 15:01:56.949143 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:56.949145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:56.949329 (+ 184us) server_negotiation.cc:407] Connection header received
0925 15:01:56.949819 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:56.949820 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:56.949832 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:56.949884 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:56.950802 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.951284 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.952311 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.952496 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.952551 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.954648 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:56.954650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:56.954650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:56.954657 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:56.955857 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.956312 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.956313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.956314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.956378 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:56.956693 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:56.956693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:56.956694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:56.956827 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:56.956891 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:56.957196 (+ 305us) server_negotiation.cc:299] Negotiation successful
0925 15:01:56.957266 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:01:56.957556 26669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:56.949053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39845 (local address 127.8.13.194:40439)
0925 15:01:56.949219 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:56.949254 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:56.949727 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:56.949954 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:56.949959 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:56.950513 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:56.950709 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.950714 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.951365 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.951366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.952231 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:56.952234 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:56.952563 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:56.952564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:56.952688 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:56.953192 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:56.953200 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:56.954557 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:56.955957 (+ 1400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.955963 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.955964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.956234 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.956477 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:56.956478 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:56.956479 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:56.956635 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:56.956888 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:56.956893 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:56.957056 (+ 163us) client_negotiation.cc:769] Sending connection context
0925 15:01:56.957169 (+ 113us) client_negotiation.cc:240] Negotiation successful
0925 15:01:56.957334 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260925 15:01:56.958034 26667 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39845
I20260925 15:01:56.958129 26667 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:56.958318 26667 heartbeater.cc:511] Master 127.8.13.254:39845 requested a full tablet report, sending...
I20260925 15:01:56.958719 26342 ts_manager.cc:194] Registered new tserver with Master: 075dc09318b249a1b7e50092c576a75a (127.8.13.194:34961)
I20260925 15:01:56.959088 26342 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:40439
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:01:56.979454 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:44375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:57.059667 26672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:57.059870 26672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:57.059893 26672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:57.061117 26672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:57.061170 26672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:57.061196 26672 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:57.062664 26672 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:44375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:57.062901 26672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:57.063122 26672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:57.067915 26680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.068068 26672 server_base.cc:1034] running on GCE node
W20260925 15:01:57.067898 26677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.067886 26678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.068351 26672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:57.068567 26672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:57.069701 26672 hybrid_clock.cc:648] HybridClock initialized: now 1790348517069689 us; error 29 us; skew 500 ppm
Sep 25 15:01:57 dist-test-slave-15hz krb5kdc[26297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348517, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:57.072186 26672 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:01:57.072614 26672 webserver.cc:460] Webserver started at http://127.8.13.195:39717/ using document root <none> and password file <none>
I20260925 15:01:57.072808 26672 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:57.072845 26672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:57.072942 26672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:57.073963 26672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "c488b957d3f44bab8d9e22f2ad11927e"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "55edb9d826382cab5776544e875fb45d"
server_key_iv: "4bd8fb2dcea347db7bbe4a7e4eb26f0e"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.074265 26672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "c488b957d3f44bab8d9e22f2ad11927e"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "55edb9d826382cab5776544e875fb45d"
server_key_iv: "4bd8fb2dcea347db7bbe4a7e4eb26f0e"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.075474 26672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:57.076196 26687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.076380 26672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:57.076467 26672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "c488b957d3f44bab8d9e22f2ad11927e"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "55edb9d826382cab5776544e875fb45d"
server_key_iv: "4bd8fb2dcea347db7bbe4a7e4eb26f0e"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.076534 26672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:57.096745 26672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:57.098712 26672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:57.098835 26672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:57.099051 26672 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:57.099349 26672 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:57.099381 26672 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.099402 26672 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:57.099426 26672 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.104902 26672 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:43111
I20260925 15:01:57.104975 26800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:43111 every 8 connection(s)
I20260925 15:01:57.105278 26672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:01:57 dist-test-slave-15hz krb5kdc[26297](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348517, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:57.114046 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.105966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:46449 (local address 127.8.13.254:39845)
0925 15:01:57.106032 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.106034 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.106205 (+ 171us) server_negotiation.cc:407] Connection header received
0925 15:01:57.106707 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.106709 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.106721 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.106789 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.107755 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.108252 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.109218 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.109416 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.109471 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.111501 (+ 2030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.111503 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.111503 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:57.111511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.112738 (+ 1227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.113102 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.113103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.113104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.113158 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.113377 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.113378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.113378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.113493 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.113551 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.113807 (+ 256us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.113928 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260925 15:01:57.114161 26803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.105927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:39845 (local address 127.8.13.195:46449)
0925 15:01:57.106094 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.106131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.106615 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.106859 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.106862 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.107420 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:57.107664 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.107670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.108335 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.108335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.109136 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.109138 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.109485 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.109485 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.109608 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.110063 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:57.110071 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:57.111411 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:57.112809 (+ 1398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:57.112817 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:57.112820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:57.113036 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:57.113212 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:57.113212 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:57.113213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:57.113322 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:57.113545 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:57.113552 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:57.113690 (+ 138us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.113780 (+ 90us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.113941 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260925 15:01:57.114216 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26672
I20260925 15:01:57.114323 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:57.114523 8247 external_mini_cluster.cc:1428] Setting key 7fc793f20c1206817d5c7e64ad759e77
I20260925 15:01:57.114539 26801 heartbeater.cc:347] Connected to a master server at 127.8.13.254:39845
I20260925 15:01:57.114637 26801 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:57.114825 26801 heartbeater.cc:511] Master 127.8.13.254:39845 requested a full tablet report, sending...
I20260925 15:01:57.115197 26341 ts_manager.cc:194] Registered new tserver with Master: c488b957d3f44bab8d9e22f2ad11927e (127.8.13.195:43111)
I20260925 15:01:57.115624 26341 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:46449
I20260925 15:01:57.126164 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:57.133285 26381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.127466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39922 (local address 127.8.13.254:39845)
0925 15:01:57.127522 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.127549 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.127668 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:01:57.127708 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.127709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.127717 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.127781 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.128256 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.128741 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.129406 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.129580 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.129630 (+ 50us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.130891 (+ 1261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.130892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.130893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:57.130900 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.132147 (+ 1247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.132523 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.132524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.132525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.132574 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:57.132758 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:57.132759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:57.132759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:57.132892 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.132987 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.133132 (+ 145us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.133190 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:01:57.134953 26341 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39922:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:57.136847 26341 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:57.142799 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.138811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43107 (local address 127.0.0.1:52900)
0925 15:01:57.139118 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.139159 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.139239 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.139517 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.139520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.139523 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.139626 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.139628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.140554 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.140555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.141649 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.141652 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142496 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.142583 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.142597 (+ 14us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.142662 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.142714 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":59,"threads_started":1}
I20260925 15:01:57.142805 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.138931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52900 (local address 127.8.13.193:43107)
0925 15:01:57.139106 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.139111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.139230 (+ 119us) server_negotiation.cc:407] Connection header received
0925 15:01:57.139326 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.139328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.139342 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.139422 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.139704 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.140468 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.141741 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142428 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142487 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.142552 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.142681 (+ 129us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.142722 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":86,"threads_started":1}
I20260925 15:01:57.143220 26811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.138811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43111 (local address 127.0.0.1:48560)
0925 15:01:57.139294 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.139303 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.139353 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.139521 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.139522 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.139555 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.139698 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.139701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.141066 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.141067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.142161 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.142163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142996 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.143065 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.143069 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.143116 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.143152 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":31,"thread_start_us":53,"threads_started":1}
I20260925 15:01:57.143252 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.139036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48560 (local address 127.8.13.195:43111)
0925 15:01:57.139200 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.139203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.139357 (+ 154us) server_negotiation.cc:407] Connection header received
0925 15:01:57.139419 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.139420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.139434 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.139504 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.139860 (+ 356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.140957 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142241 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142919 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142981 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.143032 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.143128 (+ 96us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.143168 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":81,"threads_started":1}
I20260925 15:01:57.143640 26810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.138820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:34961 (local address 127.0.0.1:54218)
0925 15:01:57.139345 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.139353 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.139392 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.139748 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.139749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.139751 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.139794 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.139795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.141073 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.141074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.141956 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.141958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142814 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.143219 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.143226 (+ 7us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.143499 (+ 273us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.143571 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":30,"thread_start_us":50,"threads_started":1}
I20260925 15:01:57.143556 26467 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 (DEFAULT_TABLE table=test-table [id=df017ecb34e54a049d67c49a557aec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.143726 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.139050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54218 (local address 127.8.13.194:34961)
0925 15:01:57.139255 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.139259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.139439 (+ 180us) server_negotiation.cc:407] Connection header received
0925 15:01:57.139478 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.139480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.139495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.139675 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.139863 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.140963 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142026 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.142665 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.142729 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.142780 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.143599 (+ 819us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.143651 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":73,"threads_started":1}
I20260925 15:01:57.143949 26467 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.143908 26735 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 (DEFAULT_TABLE table=test-table [id=df017ecb34e54a049d67c49a557aec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.144191 26735 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.144668 26601 tablet_service.cc:1467] Processing CreateTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 (DEFAULT_TABLE table=test-table [id=df017ecb34e54a049d67c49a557aec41]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.144937 26601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.147431 26815 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: Bootstrap starting.
I20260925 15:01:57.147673 26816 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Bootstrap starting.
I20260925 15:01:57.148036 26815 tablet_bootstrap.cc:654] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.148226 26816 tablet_bootstrap.cc:654] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.148525 26815 log.cc:824] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.148689 26816 log.cc:824] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.148993 26817 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Bootstrap starting.
I20260925 15:01:57.149307 26816 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: No bootstrap required, opened a new log
I20260925 15:01:57.149363 26816 ts_tablet_manager.cc:1397] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:57.149433 26815 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: No bootstrap required, opened a new log
I20260925 15:01:57.149539 26815 ts_tablet_manager.cc:1397] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:57.149593 26817 tablet_bootstrap.cc:654] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.150097 26817 log.cc:824] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.150858 26817 tablet_bootstrap.cc:492] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: No bootstrap required, opened a new log
I20260925 15:01:57.150926 26817 ts_tablet_manager.cc:1397] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:57.150843 26816 raft_consensus.cc:348] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.150970 26816 raft_consensus.cc:374] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.150997 26816 raft_consensus.cc:729] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c488b957d3f44bab8d9e22f2ad11927e, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.151085 26816 consensus_queue.cc:260] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.151335 26816 ts_tablet_manager.cc:1428] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:57.151355 26801 heartbeater.cc:503] Master 127.8.13.254:39845 was elected leader, sending a full tablet report...
I20260925 15:01:57.151995 26815 raft_consensus.cc:348] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.152240 26815 raft_consensus.cc:374] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.152319 26815 raft_consensus.cc:729] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e0b69f0bca246a2bb19d4efc4a4ab7d, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.152444 26815 consensus_queue.cc:260] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.152710 26815 ts_tablet_manager.cc:1428] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260925 15:01:57.152720 26533 heartbeater.cc:503] Master 127.8.13.254:39845 was elected leader, sending a full tablet report...
I20260925 15:01:57.152902 26817 raft_consensus.cc:348] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.153035 26817 raft_consensus.cc:374] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.153075 26817 raft_consensus.cc:729] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 075dc09318b249a1b7e50092c576a75a, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.153180 26817 consensus_queue.cc:260] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.153431 26817 ts_tablet_manager.cc:1428] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:01:57.153455 26667 heartbeater.cc:503] Master 127.8.13.254:39845 was elected leader, sending a full tablet report...
I20260925 15:01:57.161754 26822 raft_consensus.cc:482] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:57.161888 26822 raft_consensus.cc:504] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.162227 26822 leader_election.cc:288] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c488b957d3f44bab8d9e22f2ad11927e (127.8.13.195:43111), 075dc09318b249a1b7e50092c576a75a (127.8.13.194:34961)
I20260925 15:01:57.164898 26814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.162427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:47557 (local address 127.8.13.194:34961)
0925 15:01:57.162493 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.162495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.162505 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:57.162577 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.162578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.162591 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.162635 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.162770 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.163255 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164214 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.164595 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164655 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.164665 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.164792 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.164828 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260925 15:01:57.164898 26535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.162379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:34961 (local address 127.8.13.193:47557)
0925 15:01:57.162460 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.162473 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.162518 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.162651 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.162651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.162653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.162718 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.162719 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.163329 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.163330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.164131 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.164134 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.164665 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.164736 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.164739 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.164782 (+ 43us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.164824 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260925 15:01:57.164983 26813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.162461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:52915 (local address 127.8.13.195:43111)
0925 15:01:57.162513 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.162515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.162690 (+ 175us) server_negotiation.cc:407] Connection header received
0925 15:01:57.162745 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.162746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.162754 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.162805 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.162929 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.163386 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164349 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.164702 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164768 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.164776 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.164893 (+ 117us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.164926 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260925 15:01:57.165278 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8ee2a34624491bfbdb4d5ca4c4e35" candidate_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075dc09318b249a1b7e50092c576a75a" is_pre_election: true
I20260925 15:01:57.165398 26824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.162380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43111 (local address 127.8.13.193:52915)
0925 15:01:57.162624 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.162634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.162688 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.162810 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.162811 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.162812 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.162877 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.162878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.163455 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.163456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.164270 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.164272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.164769 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.164769 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.164835 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.164839 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.164883 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.164918 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":87,"thread_start_us":104,"threads_started":1}
I20260925 15:01:57.165441 26621 raft_consensus.cc:2454] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0b69f0bca246a2bb19d4efc4a4ab7d in term 0.
I20260925 15:01:57.165663 26422 leader_election.cc:302] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 075dc09318b249a1b7e50092c576a75a, 0e0b69f0bca246a2bb19d4efc4a4ab7d; no voters:
I20260925 15:01:57.165705 26755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8ee2a34624491bfbdb4d5ca4c4e35" candidate_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c488b957d3f44bab8d9e22f2ad11927e" is_pre_election: true
I20260925 15:01:57.165798 26822 raft_consensus.cc:2781] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:57.165848 26822 raft_consensus.cc:482] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:57.165844 26755 raft_consensus.cc:2454] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0e0b69f0bca246a2bb19d4efc4a4ab7d in term 0.
I20260925 15:01:57.165872 26822 raft_consensus.cc:3037] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.166658 26822 raft_consensus.cc:504] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.166781 26822 leader_election.cc:288] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [CANDIDATE]: Term 1 election: Requested vote from peers c488b957d3f44bab8d9e22f2ad11927e (127.8.13.195:43111), 075dc09318b249a1b7e50092c576a75a (127.8.13.194:34961)
I20260925 15:01:57.166913 26755 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8ee2a34624491bfbdb4d5ca4c4e35" candidate_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c488b957d3f44bab8d9e22f2ad11927e"
I20260925 15:01:57.166939 26621 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12c8ee2a34624491bfbdb4d5ca4c4e35" candidate_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "075dc09318b249a1b7e50092c576a75a"
I20260925 15:01:57.166980 26755 raft_consensus.cc:3037] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.167001 26621 raft_consensus.cc:3037] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.167677 26755 raft_consensus.cc:2454] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0b69f0bca246a2bb19d4efc4a4ab7d in term 1.
I20260925 15:01:57.167757 26621 raft_consensus.cc:2454] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0e0b69f0bca246a2bb19d4efc4a4ab7d in term 1.
I20260925 15:01:57.167857 26421 leader_election.cc:302] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0e0b69f0bca246a2bb19d4efc4a4ab7d, c488b957d3f44bab8d9e22f2ad11927e; no voters:
I20260925 15:01:57.167946 26822 raft_consensus.cc:2781] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:57.168085 26822 raft_consensus.cc:686] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [term 1 LEADER]: Becoming Leader. State: Replica: 0e0b69f0bca246a2bb19d4efc4a4ab7d, State: Running, Role: LEADER
I20260925 15:01:57.168165 26822 consensus_queue.cc:237] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } }
I20260925 15:01:57.168936 26341 catalog_manager.cc:5495] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0e0b69f0bca246a2bb19d4efc4a4ab7d (127.8.13.193). New cstate: current_term: 1 leader_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0e0b69f0bca246a2bb19d4efc4a4ab7d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:57.178363 26812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.176207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52904 (local address 127.8.13.193:43107)
0925 15:01:57.176287 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.176289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.176298 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:01:57.176331 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.176332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.176355 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.176426 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:57.176530 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.176981 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.177734 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.177912 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.177959 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.178073 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:57.178157 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:57.178222 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.178256 (+ 34us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.178267 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260925 15:01:57.180110 26621 raft_consensus.cc:1261] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 1 FOLLOWER]: Refusing update from remote peer 0e0b69f0bca246a2bb19d4efc4a4ab7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.180137 26755 raft_consensus.cc:1261] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 1 FOLLOWER]: Refusing update from remote peer 0e0b69f0bca246a2bb19d4efc4a4ab7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.180481 26825 consensus_queue.cc:1035] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c488b957d3f44bab8d9e22f2ad11927e" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.180603 26822 consensus_queue.cc:1035] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 0e0b69f0bca246a2bb19d4efc4a4ab7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "075dc09318b249a1b7e50092c576a75a" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 34961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.183172 26829 mvcc.cc:205] Tried to move back new op lower bound from 7333267526367195136 to 7333267526321455104. Current Snapshot: MvccSnapshot[applied={T|T < 7333267526367195136}]
I20260925 15:01:57.183375 26828 mvcc.cc:205] Tried to move back new op lower bound from 7333267526367195136 to 7333267526321455104. Current Snapshot: MvccSnapshot[applied={T|T < 7333267526367195136 or (T in {7333267526367195136})}]
I20260925 15:01:57.184947 26342 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.185024 26342 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39922:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.185711 26342 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a319b3acd53e4902b1c7664ab96d4561: Sending DeleteTablet for 3 replicas of tablet 12c8ee2a34624491bfbdb4d5ca4c4e35
I20260925 15:01:57.186058 26467 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:52900
I20260925 15:01:57.186115 26735 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:48560
I20260925 15:01:57.186120 26601 tablet_service.cc:1514] Processing DeleteTablet for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:54218
I20260925 15:01:57.186332 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26404
I20260925 15:01:57.186350 26839 tablet_replica.cc:330] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: stopping tablet replica
I20260925 15:01:57.186976 26839 raft_consensus.cc:2229] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:57.187099 26839 raft_consensus.cc:2258] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.187119 26838 tablet_replica.cc:330] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: stopping tablet replica
I20260925 15:01:57.187247 26838 raft_consensus.cc:2229] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:57.187373 26838 raft_consensus.cc:2258] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.187640 26839 ts_tablet_manager.cc:1905] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.187810 26838 ts_tablet_manager.cc:1905] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.189260 26839 ts_tablet_manager.cc:1918] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.189282 26838 ts_tablet_manager.cc:1918] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.189328 26839 log.cc:1196] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/12c8ee2a34624491bfbdb4d5ca4c4e35
I20260925 15:01:57.189337 26838 log.cc:1196] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/12c8ee2a34624491bfbdb4d5ca4c4e35
I20260925 15:01:57.189550 26839 ts_tablet_manager.cc:1939] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P 075dc09318b249a1b7e50092c576a75a: Deleting consensus metadata
I20260925 15:01:57.189577 26838 ts_tablet_manager.cc:1939] T 12c8ee2a34624491bfbdb4d5ca4c4e35 P c488b957d3f44bab8d9e22f2ad11927e: Deleting consensus metadata
I20260925 15:01:57.189911 26328 catalog_manager.cc:4841] TS 075dc09318b249a1b7e50092c576a75a (127.8.13.194:34961): tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 (table test-table [id=df017ecb34e54a049d67c49a557aec41]) successfully deleted
I20260925 15:01:57.190058 26327 catalog_manager.cc:4841] TS c488b957d3f44bab8d9e22f2ad11927e (127.8.13.195:43111): tablet 12c8ee2a34624491bfbdb4d5ca4c4e35 (table test-table [id=df017ecb34e54a049d67c49a557aec41]) successfully deleted
W20260925 15:01:57.192377 26329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:43107 (error 108)
I20260925 15:01:57.192667 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26538
I20260925 15:01:57.192795 26809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.192677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43107 (local address 127.0.0.1:52906)
0925 15:01:57.192731 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.192763 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43107: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:01:57.193253 26329 catalog_manager.cc:4584] TS 0e0b69f0bca246a2bb19d4efc4a4ab7d (127.8.13.193:43107): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 12c8ee2a34624491bfbdb4d5ca4c4e35: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43107: connect: Connection refused (error 111)
I20260925 15:01:57.197445 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26672
I20260925 15:01:57.201540 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26310
2026-09-25T15:01:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1769 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-25T15:01:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:57Z Disabled control of system clock
I20260925 15:01:57.227468 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46735
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:57.310494 26844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:57.310729 26844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:57.310750 26844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:57.312001 26844 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:57.312052 26844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:57.312063 26844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:57.312074 26844 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:57.312086 26844 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:57.313465 26844 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46735
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:57.313673 26844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:57.313930 26844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:57.318603 26844 server_base.cc:1034] running on GCE node
W20260925 15:01:57.318574 26852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.318559 26849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.318571 26850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.319010 26844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:57.319301 26844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:57.320434 26844 hybrid_clock.cc:648] HybridClock initialized: now 1790348517320416 us; error 34 us; skew 500 ppm
I20260925 15:01:57.321640 26844 webserver.cc:460] Webserver started at http://127.8.13.254:33465/ using document root <none> and password file <none>
I20260925 15:01:57.321895 26844 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:57.321941 26844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:57.322072 26844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:57.323151 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "98226f08e4304d339a38d0e31126ac70"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "dea77785ad2ae43c77eb4db121d44c93"
server_key_iv: "683032a89d27d868fc3a37851e824e28"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.323495 26844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "98226f08e4304d339a38d0e31126ac70"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "dea77785ad2ae43c77eb4db121d44c93"
server_key_iv: "683032a89d27d868fc3a37851e824e28"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.324820 26844 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:01:57.325564 26858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.325706 26844 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:57.325794 26844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "98226f08e4304d339a38d0e31126ac70"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "dea77785ad2ae43c77eb4db121d44c93"
server_key_iv: "683032a89d27d868fc3a37851e824e28"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.325867 26844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:57.335914 26844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:57.336233 26844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:57.336365 26844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:57.340032 26844 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:46735
I20260925 15:01:57.340068 26910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:46735 every 8 connection(s)
I20260925 15:01:57.340384 26844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:57.341372 26911 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.341444 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26844
I20260925 15:01:57.341544 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:57.341748 8247 external_mini_cluster.cc:1428] Setting key f48d5daf8700ce165dc1679b0bfe66b9
I20260925 15:01:57.344026 26911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: Bootstrap starting.
I20260925 15:01:57.344642 26911 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.344920 26911 log.cc:824] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.345542 26911 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: No bootstrap required, opened a new log
I20260925 15:01:57.347357 26911 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } }
I20260925 15:01:57.347514 26911 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.347594 26911 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98226f08e4304d339a38d0e31126ac70, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.347730 26911 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } }
I20260925 15:01:57.347795 26911 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:57.347818 26911 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:57.347863 26911 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.348196 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.342289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45082 (local address 127.8.13.254:46735)
0925 15:01:57.342477 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.342486 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.343792 (+ 1306us) server_negotiation.cc:407] Connection header received
0925 15:01:57.344272 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.344279 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.344343 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.344457 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.344678 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.345602 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.346342 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.346745 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.346853 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.347548 (+ 695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.347553 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.347553 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:57.347756 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.347799 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:57.347814 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.347868 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.347951 (+ 83us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.348033 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
I20260925 15:01:57.348510 26911 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } }
I20260925 15:01:57.348618 26911 leader_election.cc:302] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98226f08e4304d339a38d0e31126ac70; no voters:
I20260925 15:01:57.348767 26911 leader_election.cc:288] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:57.348803 26916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:57.348941 26916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [term 1 LEADER]: Becoming Leader. State: Replica: 98226f08e4304d339a38d0e31126ac70, State: Running, Role: LEADER
I20260925 15:01:57.349009 26911 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:57.349022 26916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } }
I20260925 15:01:57.349424 26917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98226f08e4304d339a38d0e31126ac70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } } }
I20260925 15:01:57.349668 26917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:57.349602 26918 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98226f08e4304d339a38d0e31126ac70. Latest consensus state: current_term: 1 leader_uuid: "98226f08e4304d339a38d0e31126ac70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98226f08e4304d339a38d0e31126ac70" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46735 } } }
I20260925 15:01:57.349782 26918 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:57.349982 26926 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:57.350381 26926 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:57.351792 26926 catalog_manager.cc:1347] Generated new cluster ID: 86ede07d90684a3c96b8cfd6b32362a8
I20260925 15:01:57.351841 26926 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:57.365300 26926 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:57.365847 26926 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:57.372267 26926 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: Generated new TSK 0
I20260925 15:01:57.372454 26926 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:57.383257 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--builtin_ntp_servers=127.8.13.212:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:57.464092 26935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:57.464313 26935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:57.464344 26935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:57.465592 26935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:57.465643 26935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:57.465679 26935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:57.467324 26935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:57.467607 26935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:57.467913 26935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:57.472829 26940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.472932 26943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.472800 26941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.473104 26935 server_base.cc:1034] running on GCE node
I20260925 15:01:57.473287 26935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:57.473495 26935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:57.474677 26935 hybrid_clock.cc:648] HybridClock initialized: now 1790348517474662 us; error 27 us; skew 500 ppm
I20260925 15:01:57.476081 26935 webserver.cc:460] Webserver started at http://127.8.13.193:33727/ using document root <none> and password file <none>
I20260925 15:01:57.476305 26935 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:57.476351 26935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:57.476468 26935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:57.477566 26935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "15c3d7ab717f408e999ebcdea29dc8c2"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "91ae14dad8c09985c0e22234b1523007"
server_key_iv: "d33eeca0f06dde1d5b173ceea1d99dea"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.477910 26935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "15c3d7ab717f408e999ebcdea29dc8c2"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "91ae14dad8c09985c0e22234b1523007"
server_key_iv: "d33eeca0f06dde1d5b173ceea1d99dea"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.479136 26935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:57.479869 26949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.480005 26935 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.480085 26935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "15c3d7ab717f408e999ebcdea29dc8c2"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "91ae14dad8c09985c0e22234b1523007"
server_key_iv: "d33eeca0f06dde1d5b173ceea1d99dea"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.480146 26935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:57.489569 26935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:57.489841 26935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:57.489971 26935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:57.490167 26935 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:57.490460 26935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:57.490494 26935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.490530 26935 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:57.490553 26935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.496291 26935 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:43313
I20260925 15:01:57.496342 27062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:43313 every 8 connection(s)
I20260925 15:01:57.496659 26935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:57.497861 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 26935
I20260925 15:01:57.497962 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:57.498159 8247 external_mini_cluster.cc:1428] Setting key bb843ef0f2eab3afeac8081e9b781a2d
I20260925 15:01:57.499111 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--builtin_ntp_servers=127.8.13.212:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:01:57.502435 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.497301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:39321 (local address 127.8.13.254:46735)
0925 15:01:57.497360 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.497362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.497590 (+ 228us) server_negotiation.cc:407] Connection header received
0925 15:01:57.498262 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.498263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.498280 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.498353 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.499202 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.499791 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.500920 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.501216 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.501287 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.502159 (+ 872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.502160 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.502160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:57.502165 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.502183 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:57.502188 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.502245 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.502319 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.502347 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:01:57.502638 27065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.497275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46735 (local address 127.8.13.193:39321)
0925 15:01:57.497466 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.497512 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.498143 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.498444 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.498449 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.498459 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:57.499109 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.499117 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.499870 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.499872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.500824 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.500827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.501318 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.501319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.501435 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.501981 (+ 546us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:57.501982 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:57.502012 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:57.502056 (+ 44us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:57.502251 (+ 195us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:57.502254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:57.502257 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.502308 (+ 51us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.502393 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 15:01:57.503140 27063 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46735
I20260925 15:01:57.503243 27063 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:57.503428 27063 heartbeater.cc:511] Master 127.8.13.254:46735 requested a full tablet report, sending...
I20260925 15:01:57.504025 26875 ts_manager.cc:194] Registered new tserver with Master: 15c3d7ab717f408e999ebcdea29dc8c2 (127.8.13.193:43313)
I20260925 15:01:57.504647 26875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.193:39321
W20260925 15:01:57.579700 27066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:57.579905 27066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:57.579926 27066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:57.581131 27066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:57.581174 27066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:57.581200 27066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:57.582602 27066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27066
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:57.582834 27066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:57.583112 27066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:57.587836 27071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.587857 27074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.587916 27072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.588032 27066 server_base.cc:1034] running on GCE node
I20260925 15:01:57.588265 27066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:57.588505 27066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:57.589634 27066 hybrid_clock.cc:648] HybridClock initialized: now 1790348517589624 us; error 28 us; skew 500 ppm
I20260925 15:01:57.590883 27066 webserver.cc:460] Webserver started at http://127.8.13.194:39217/ using document root <none> and password file <none>
I20260925 15:01:57.591117 27066 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:57.591171 27066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:57.591297 27066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:57.592551 27066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "8727d94b69fb449faa5d44dd5e6b48ed"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "394b7811f818ec1baa4792665dfa5bbe"
server_key_iv: "61ca1c6fe17f6cf3af40130fb7155e92"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.592934 27066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "8727d94b69fb449faa5d44dd5e6b48ed"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "394b7811f818ec1baa4792665dfa5bbe"
server_key_iv: "61ca1c6fe17f6cf3af40130fb7155e92"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.594112 27066 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260925 15:01:57.594730 27080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.594875 27066 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:57.594962 27066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "8727d94b69fb449faa5d44dd5e6b48ed"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "394b7811f818ec1baa4792665dfa5bbe"
server_key_iv: "61ca1c6fe17f6cf3af40130fb7155e92"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.595031 27066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:57.619683 27066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:57.619984 27066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:57.620126 27066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:57.620338 27066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:57.620667 27066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:57.620700 27066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.620730 27066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:57.620747 27066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.625964 27066 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:44613
I20260925 15:01:57.626015 27193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:44613 every 8 connection(s)
I20260925 15:01:57.626320 27066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:57.631208 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.626992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:55171 (local address 127.8.13.254:46735)
0925 15:01:57.627045 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.627047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.627274 (+ 227us) server_negotiation.cc:407] Connection header received
0925 15:01:57.627799 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.627800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.627810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.627866 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.628590 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.629078 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.630048 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.630255 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.630303 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.631008 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.631009 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.631009 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:57.631014 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.631029 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:57.631032 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.631070 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.631123 (+ 53us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.631145 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:01:57.631387 27196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.626965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46735 (local address 127.8.13.194:55171)
0925 15:01:57.627155 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.627195 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.627709 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.627931 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.627935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.627943 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:57.628494 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.628500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.629182 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.629183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.629974 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.629976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.630319 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.630319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.630425 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.630890 (+ 465us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:57.630890 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:57.630918 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:57.630939 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:57.631075 (+ 136us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:57.631077 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:57.631080 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.631124 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.631200 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260925 15:01:57.631805 27194 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46735
I20260925 15:01:57.631887 27194 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:57.632025 27194 heartbeater.cc:511] Master 127.8.13.254:46735 requested a full tablet report, sending...
I20260925 15:01:57.632357 26875 ts_manager.cc:194] Registered new tserver with Master: 8727d94b69fb449faa5d44dd5e6b48ed (127.8.13.194:44613)
I20260925 15:01:57.632688 26875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.194:55171
I20260925 15:01:57.633963 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27066
I20260925 15:01:57.634044 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:57.634202 8247 external_mini_cluster.cc:1428] Setting key 1361523bd232c631806db84c77d07194
I20260925 15:01:57.635021 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--builtin_ntp_servers=127.8.13.212:33333
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:57.713653 27197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:57.713876 27197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:57.713898 27197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:57.715061 27197 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:57.715113 27197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:57.715149 27197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:57.716612 27197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33333
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:46735
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27197
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:57.716874 27197 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:57.717180 27197 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:57.722095 27203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.722245 27202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:57.722254 27205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:57.722393 27197 server_base.cc:1034] running on GCE node
I20260925 15:01:57.722807 27197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:57.723070 27197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:57.724233 27197 hybrid_clock.cc:648] HybridClock initialized: now 1790348517724216 us; error 30 us; skew 500 ppm
I20260925 15:01:57.725502 27197 webserver.cc:460] Webserver started at http://127.8.13.195:41879/ using document root <none> and password file <none>
I20260925 15:01:57.725725 27197 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:57.725770 27197 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:57.725900 27197 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:57.726985 27197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "a5d9bc937e7f4b11bb35454929fc8968"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "541b8c31f489314ff8a049a14f82c10e"
server_key_iv: "b2ff3594668b1a725d0ecfdb60e4b240"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.727332 27197 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "a5d9bc937e7f4b11bb35454929fc8968"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "541b8c31f489314ff8a049a14f82c10e"
server_key_iv: "b2ff3594668b1a725d0ecfdb60e4b240"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.728672 27197 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:57.729367 27211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.729540 27197 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:57.729619 27197 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "a5d9bc937e7f4b11bb35454929fc8968"
format_stamp: "Formatted at 2026-09-25 15:01:57 on dist-test-slave-15hz"
server_key: "541b8c31f489314ff8a049a14f82c10e"
server_key_iv: "b2ff3594668b1a725d0ecfdb60e4b240"
server_key_version: "encryptionkey@0"
I20260925 15:01:57.729692 27197 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:57.744992 27197 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:57.745298 27197 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:57.745431 27197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:57.745631 27197 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:57.745935 27197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:57.745967 27197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.745990 27197 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:57.746012 27197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:57.751523 27197 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:37935
I20260925 15:01:57.751597 27324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:37935 every 8 connection(s)
I20260925 15:01:57.751914 27197 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:01:57.756973 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.752632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:54447 (local address 127.8.13.254:46735)
0925 15:01:57.752720 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.752722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.752904 (+ 182us) server_negotiation.cc:407] Connection header received
0925 15:01:57.753414 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.753415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.753427 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.753475 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.754163 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.754653 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.755666 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.755855 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.755910 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.756713 (+ 803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.756714 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.756714 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:57.756720 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.756737 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:57.756741 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.756783 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.756857 (+ 74us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.756884 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260925 15:01:57.757179 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.752572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:46735 (local address 127.8.13.195:54447)
0925 15:01:57.752787 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.752823 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.753313 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.753559 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.753563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.753572 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:57.754069 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.754075 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.754751 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.754752 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.755579 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.755582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.755925 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.755926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.756056 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.756581 (+ 525us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:01:57.756582 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:57.756611 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:01:57.756636 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:57.756789 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:57.756794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:57.756797 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.756855 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.756961 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":90,"threads_started":1}
I20260925 15:01:57.757687 27325 heartbeater.cc:347] Connected to a master server at 127.8.13.254:46735
I20260925 15:01:57.757795 27325 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:57.757982 27325 heartbeater.cc:511] Master 127.8.13.254:46735 requested a full tablet report, sending...
I20260925 15:01:57.758355 26875 ts_manager.cc:194] Registered new tserver with Master: a5d9bc937e7f4b11bb35454929fc8968 (127.8.13.195:37935)
I20260925 15:01:57.758724 26875 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.195:54447
I20260925 15:01:57.759326 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27197
I20260925 15:01:57.759408 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:57.759665 8247 external_mini_cluster.cc:1428] Setting key 7e31a61bdea31b65d28a638b65a8eb24
I20260925 15:01:57.760669 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:57.764940 26914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.761986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45092 (local address 127.8.13.254:46735)
0925 15:01:57.762071 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.762073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.762136 (+ 63us) server_negotiation.cc:407] Connection header received
0925 15:01:57.762188 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.762189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.762198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.762242 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:57.762357 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.762826 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.763582 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.763797 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.763852 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.764538 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:57.764539 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:57.764540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:57.764545 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:57.764560 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:57.764564 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:57.764656 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.764831 (+ 175us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.764846 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:01:57.766518 26875 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45092:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:57.768090 26875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:57.774442 27333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43313 (local address 127.0.0.1:53342)
0925 15:01:57.770666 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.770677 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.770726 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.770956 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.770959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.770961 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.771045 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.771046 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772168 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.772169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.773273 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.773275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774133 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.774210 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774223 (+ 13us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.774293 (+ 70us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.774348 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":127,"threads_started":1}
I20260925 15:01:57.774538 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40710 (local address 127.8.13.195:37935)
0925 15:01:57.771114 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.771118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.771126 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:01:57.771168 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.771170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.771183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.771244 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.771408 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772570 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.773601 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774229 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774292 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774381 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.774421 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.774457 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":94,"threads_started":1}
I20260925 15:01:57.774814 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.8.13.193:43313)
0925 15:01:57.770799 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.770804 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.770815 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:01:57.770854 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.770855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.770868 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.770944 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.771117 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772084 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.773383 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774058 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774126 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774220 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.774693 (+ 473us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.774736 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":77,"threads_started":1}
I20260925 15:01:57.774878 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44613 (local address 127.0.0.1:37868)
0925 15:01:57.770940 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.770948 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.770998 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.771283 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.771284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.771286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.771328 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.771330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772646 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.772647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.773746 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.773749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774667 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.774720 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774723 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.774770 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.774808 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":31,"thread_start_us":62,"threads_started":1}
I20260925 15:01:57.774901 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37868 (local address 127.8.13.194:44613)
0925 15:01:57.770938 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.770943 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.770989 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:01:57.771057 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.771059 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.771074 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.771219 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.771404 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772550 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.773834 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774495 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774553 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774633 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.774796 (+ 163us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.774832 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":68,"threads_started":1}
I20260925 15:01:57.775206 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.770371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:37935 (local address 127.0.0.1:40710)
0925 15:01:57.770557 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.770600 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.770673 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.771302 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.771303 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.771304 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.771339 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.771340 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.772652 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.772653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.773528 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.773531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.774305 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.774306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.774359 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.774363 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.774403 (+ 40us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.774433 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":38,"thread_start_us":61,"threads_started":1}
I20260925 15:01:57.775554 27128 tablet_service.cc:1467] Processing CreateTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 (DEFAULT_TABLE table=test-table [id=13a64cb08c1740babf8e4ba1baf57bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.775555 26997 tablet_service.cc:1467] Processing CreateTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 (DEFAULT_TABLE table=test-table [id=13a64cb08c1740babf8e4ba1baf57bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.775913 26997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26721f4fb9de4b4daf80647a912c69e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.775913 27128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26721f4fb9de4b4daf80647a912c69e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.775934 27259 tablet_service.cc:1467] Processing CreateTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 (DEFAULT_TABLE table=test-table [id=13a64cb08c1740babf8e4ba1baf57bb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:57.776216 27259 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 26721f4fb9de4b4daf80647a912c69e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:57.779048 27339 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Bootstrap starting.
I20260925 15:01:57.779256 27340 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Bootstrap starting.
I20260925 15:01:57.779685 27341 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: Bootstrap starting.
I20260925 15:01:57.779714 27339 tablet_bootstrap.cc:654] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.779842 27340 tablet_bootstrap.cc:654] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.780014 27339 log.cc:824] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.780154 27340 log.cc:824] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.780288 27341 tablet_bootstrap.cc:654] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:57.780576 27341 log.cc:824] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:57.780771 27339 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: No bootstrap required, opened a new log
I20260925 15:01:57.780807 27340 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: No bootstrap required, opened a new log
I20260925 15:01:57.780854 27340 ts_tablet_manager.cc:1397] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:01:57.780848 27339 ts_tablet_manager.cc:1397] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:57.781188 27341 tablet_bootstrap.cc:492] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: No bootstrap required, opened a new log
I20260925 15:01:57.781244 27341 ts_tablet_manager.cc:1397] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:57.783109 27341 raft_consensus.cc:348] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783248 27341 raft_consensus.cc:374] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.783281 27341 raft_consensus.cc:729] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15c3d7ab717f408e999ebcdea29dc8c2, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.783414 27341 consensus_queue.cc:260] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783421 27340 raft_consensus.cc:348] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783421 27339 raft_consensus.cc:348] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783618 27339 raft_consensus.cc:374] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.783618 27340 raft_consensus.cc:374] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:57.783669 27340 raft_consensus.cc:729] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8727d94b69fb449faa5d44dd5e6b48ed, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.783669 27339 raft_consensus.cc:729] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5d9bc937e7f4b11bb35454929fc8968, State: Initialized, Role: FOLLOWER
I20260925 15:01:57.783715 27341 ts_tablet_manager.cc:1428] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260925 15:01:57.783810 27340 consensus_queue.cc:260] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783810 27339 consensus_queue.cc:260] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.783881 27063 heartbeater.cc:503] Master 127.8.13.254:46735 was elected leader, sending a full tablet report...
I20260925 15:01:57.784076 27339 ts_tablet_manager.cc:1428] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260925 15:01:57.784076 27340 ts_tablet_manager.cc:1428] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:01:57.784211 27325 heartbeater.cc:503] Master 127.8.13.254:46735 was elected leader, sending a full tablet report...
I20260925 15:01:57.784271 27194 heartbeater.cc:503] Master 127.8.13.254:46735 was elected leader, sending a full tablet report...
W20260925 15:01:57.876884 27195 tablet.cc:2369] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:57.903457 27347 raft_consensus.cc:482] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:57.903633 27347 raft_consensus.cc:504] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.904031 27347 leader_election.cc:288] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15c3d7ab717f408e999ebcdea29dc8c2 (127.8.13.193:43313), 8727d94b69fb449faa5d44dd5e6b48ed (127.8.13.194:44613)
I20260925 15:01:57.906709 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.904299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:45975 (local address 127.8.13.194:44613)
0925 15:01:57.904346 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.904348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.904365 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:01:57.904399 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.904400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.904408 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.904453 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.904605 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.905152 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906060 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.906401 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906462 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.906472 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.906603 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.906641 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260925 15:01:57.906709 27327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.904220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44613 (local address 127.8.13.195:45975)
0925 15:01:57.904276 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.904288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.904332 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.904470 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.904470 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.904472 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.904542 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.904544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.905237 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.905238 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.905963 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.905965 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.906476 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.906539 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.906542 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.906586 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.906631 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:01:57.906778 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.904283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:57307 (local address 127.8.13.193:43313)
0925 15:01:57.904331 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.904334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.904467 (+ 133us) server_negotiation.cc:407] Connection header received
0925 15:01:57.904514 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.904515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.904526 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.904566 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:57.904709 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.905244 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906128 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.906480 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906538 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.906546 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.906677 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.906715 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:01:57.906992 27348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.904230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43313 (local address 127.8.13.195:57307)
0925 15:01:57.904394 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.904403 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:57.904461 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:57.904581 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:57.904581 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:57.904583 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:57.904646 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.904647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.905313 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.905314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.906044 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:57.906045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.906552 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.906553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:57.906615 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.906617 (+ 2us) client_negotiation.cc:769] Sending connection context
0925 15:01:57.906661 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:01:57.906696 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":15,"thread_start_us":105,"threads_started":1}
I20260925 15:01:57.907058 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26721f4fb9de4b4daf80647a912c69e5" candidate_uuid: "a5d9bc937e7f4b11bb35454929fc8968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" is_pre_election: true
I20260925 15:01:57.907214 27148 raft_consensus.cc:2454] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d9bc937e7f4b11bb35454929fc8968 in term 0.
I20260925 15:01:57.907315 27017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26721f4fb9de4b4daf80647a912c69e5" candidate_uuid: "a5d9bc937e7f4b11bb35454929fc8968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" is_pre_election: true
I20260925 15:01:57.907456 27017 raft_consensus.cc:2454] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a5d9bc937e7f4b11bb35454929fc8968 in term 0.
I20260925 15:01:57.907693 27212 leader_election.cc:302] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 15c3d7ab717f408e999ebcdea29dc8c2, a5d9bc937e7f4b11bb35454929fc8968; no voters:
I20260925 15:01:57.907851 27347 raft_consensus.cc:2781] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:57.907903 27347 raft_consensus.cc:482] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:57.907923 27347 raft_consensus.cc:3037] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.908562 27347 raft_consensus.cc:504] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.908677 27347 leader_election.cc:288] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [CANDIDATE]: Term 1 election: Requested vote from peers 15c3d7ab717f408e999ebcdea29dc8c2 (127.8.13.193:43313), 8727d94b69fb449faa5d44dd5e6b48ed (127.8.13.194:44613)
I20260925 15:01:57.908846 27017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26721f4fb9de4b4daf80647a912c69e5" candidate_uuid: "a5d9bc937e7f4b11bb35454929fc8968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15c3d7ab717f408e999ebcdea29dc8c2"
I20260925 15:01:57.908878 27148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "26721f4fb9de4b4daf80647a912c69e5" candidate_uuid: "a5d9bc937e7f4b11bb35454929fc8968" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8727d94b69fb449faa5d44dd5e6b48ed"
I20260925 15:01:57.908921 27017 raft_consensus.cc:3037] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.908942 27148 raft_consensus.cc:3037] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:57.909619 27148 raft_consensus.cc:2454] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d9bc937e7f4b11bb35454929fc8968 in term 1.
I20260925 15:01:57.909662 27017 raft_consensus.cc:2454] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a5d9bc937e7f4b11bb35454929fc8968 in term 1.
I20260925 15:01:57.909795 27213 leader_election.cc:302] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8727d94b69fb449faa5d44dd5e6b48ed, a5d9bc937e7f4b11bb35454929fc8968; no voters:
I20260925 15:01:57.909888 27347 raft_consensus.cc:2781] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:57.910018 27347 raft_consensus.cc:686] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 1 LEADER]: Becoming Leader. State: Replica: a5d9bc937e7f4b11bb35454929fc8968, State: Running, Role: LEADER
I20260925 15:01:57.910100 27347 consensus_queue.cc:237] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } }
I20260925 15:01:57.910899 26874 catalog_manager.cc:5495] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 reported cstate change: term changed from 0 to 1, leader changed from <none> to a5d9bc937e7f4b11bb35454929fc8968 (127.8.13.195). New cstate: current_term: 1 leader_uuid: "a5d9bc937e7f4b11bb35454929fc8968" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a5d9bc937e7f4b11bb35454929fc8968" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 37935 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:57.921748 27336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.919700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40718 (local address 127.8.13.195:37935)
0925 15:01:57.919742 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:57.919743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:57.919753 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:01:57.919813 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:57.919814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:57.919831 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:57.919881 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:57.919983 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.920467 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.921161 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:57.921347 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:57.921389 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:57.921486 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:57.921567 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:57.921630 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:57.921671 (+ 41us) server_negotiation.cc:299] Negotiation successful
0925 15:01:57.921680 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:01:57.923396 27017 raft_consensus.cc:1261] T 26721f4fb9de4b4daf80647a912c69e5 P 15c3d7ab717f408e999ebcdea29dc8c2 [term 1 FOLLOWER]: Refusing update from remote peer a5d9bc937e7f4b11bb35454929fc8968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.923396 27148 raft_consensus.cc:1261] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 1 FOLLOWER]: Refusing update from remote peer a5d9bc937e7f4b11bb35454929fc8968: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:57.923734 27349 consensus_queue.cc:1035] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8727d94b69fb449faa5d44dd5e6b48ed" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.923869 27347 consensus_queue.cc:1035] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [LEADER]: Connected to new peer: Peer: permanent_uuid: "15c3d7ab717f408e999ebcdea29dc8c2" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 43313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:57.926083 27352 mvcc.cc:205] Tried to move back new op lower bound from 7333267529411932160 to 7333267529360465920. Current Snapshot: MvccSnapshot[applied={T|T < 7333267529411932160}]
I20260925 15:01:57.926688 27354 mvcc.cc:205] Tried to move back new op lower bound from 7333267529411932160 to 7333267529360465920. Current Snapshot: MvccSnapshot[applied={T|T < 7333267529411932160}]
I20260925 15:01:57.928447 26874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.928529 26874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45092:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:57.929278 26874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 98226f08e4304d339a38d0e31126ac70: Sending DeleteTablet for 3 replicas of tablet 26721f4fb9de4b4daf80647a912c69e5
I20260925 15:01:57.929618 27128 tablet_service.cc:1514] Processing DeleteTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='slave'} at 127.0.0.1:37868
I20260925 15:01:57.929623 26997 tablet_service.cc:1514] Processing DeleteTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='slave'} at 127.0.0.1:53342
I20260925 15:01:57.929836 27259 tablet_service.cc:1514] Processing DeleteTablet for tablet 26721f4fb9de4b4daf80647a912c69e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:57 UTC) from {username='slave'} at 127.0.0.1:40710
I20260925 15:01:57.929888 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26935
I20260925 15:01:57.930840 27364 tablet_replica.cc:330] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: stopping tablet replica
I20260925 15:01:57.930938 27364 raft_consensus.cc:2229] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:01:57.931051 27364 raft_consensus.cc:2258] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.931483 27364 ts_tablet_manager.cc:1905] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.932916 27362 tablet_replica.cc:330] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: stopping tablet replica
I20260925 15:01:57.933038 27362 raft_consensus.cc:2229] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:57.933166 27362 raft_consensus.cc:2258] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:57.933167 27364 ts_tablet_manager.cc:1918] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.933228 27364 log.cc:1196] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/26721f4fb9de4b4daf80647a912c69e5
I20260925 15:01:57.933462 27364 ts_tablet_manager.cc:1939] T 26721f4fb9de4b4daf80647a912c69e5 P a5d9bc937e7f4b11bb35454929fc8968: Deleting consensus metadata
I20260925 15:01:57.933570 27362 ts_tablet_manager.cc:1905] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:57.935011 27362 ts_tablet_manager.cc:1918] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:57.935086 27362 log.cc:1196] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/26721f4fb9de4b4daf80647a912c69e5
I20260925 15:01:57.935312 27362 ts_tablet_manager.cc:1939] T 26721f4fb9de4b4daf80647a912c69e5 P 8727d94b69fb449faa5d44dd5e6b48ed: Deleting consensus metadata
I20260925 15:01:57.935781 26860 catalog_manager.cc:4841] TS 8727d94b69fb449faa5d44dd5e6b48ed (127.8.13.194:44613): tablet 26721f4fb9de4b4daf80647a912c69e5 (table test-table [id=13a64cb08c1740babf8e4ba1baf57bb3]) successfully deleted
I20260925 15:01:57.936023 26860 catalog_manager.cc:4841] TS a5d9bc937e7f4b11bb35454929fc8968 (127.8.13.195:37935): tablet 26721f4fb9de4b4daf80647a912c69e5 (table test-table [id=13a64cb08c1740babf8e4ba1baf57bb3]) successfully deleted
W20260925 15:01:57.936923 26859 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:43313 (error 108)
I20260925 15:01:57.937119 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27066
I20260925 15:01:57.942211 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27197
I20260925 15:01:57.942366 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:57.942211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:43313 (local address 127.0.0.1:53356)
0925 15:01:57.942285 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:57.942332 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260925 15:01:57.943693 26859 catalog_manager.cc:4584] TS 15c3d7ab717f408e999ebcdea29dc8c2 (127.8.13.193:43313): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 26721f4fb9de4b4daf80647a912c69e5: Network error: Client connection negotiation failed: client connection to 127.8.13.193:43313: connect: Connection refused (error 111)
I20260925 15:01:57.946403 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 26844
2026-09-25T15:01:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (743 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:01:58 dist-test-slave-15hz krb5kdc[27368](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:01:58 dist-test-slave-15hz krb5kdc[27368](info): set up 2 sockets
Sep 25 15:01:58 dist-test-slave-15hz krb5kdc[27368](info): commencing operation
krb5kdc: starting...
W20260925 15:01:58.899132 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:01:58 dist-test-slave-15hz krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:01:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:01:59.012466 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33895
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:33895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:59.093381 27381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:59.093611 27381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:59.093653 27381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:59.094920 27381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:59.094971 27381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:59.094990 27381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:59.095006 27381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:59.095022 27381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:59.096731 27381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:33895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:33895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27381
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:59.096997 27381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:59.097213 27381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:59.102047 27387 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.102041 27386 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.102190 27381 server_base.cc:1034] running on GCE node
W20260925 15:01:59.102039 27389 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.102553 27381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:59.102843 27381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:59.104015 27381 hybrid_clock.cc:648] HybridClock initialized: now 1790348519103986 us; error 38 us; skew 500 ppm
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:59.106537 27381 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:01:59.106966 27381 webserver.cc:460] Webserver started at http://127.8.13.254:38713/ using document root <none> and password file <none>
I20260925 15:01:59.107172 27381 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:59.107223 27381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:59.107349 27381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:59.108453 27381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "47e503d1750e439eb5f9e148104ae907"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "559c06680491450d0ef40431bc2e90c5"
server_key_iv: "8bbd96760edee5ed48d591e06badd828"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.108798 27381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "47e503d1750e439eb5f9e148104ae907"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "559c06680491450d0ef40431bc2e90c5"
server_key_iv: "8bbd96760edee5ed48d591e06badd828"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.110023 27381 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:59.110671 27396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.110826 27381 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:59.110904 27381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "47e503d1750e439eb5f9e148104ae907"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "559c06680491450d0ef40431bc2e90c5"
server_key_iv: "8bbd96760edee5ed48d591e06badd828"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.110970 27381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:59.137408 27381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:59.139602 27381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:59.139741 27381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:59.143282 27381 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:33895
I20260925 15:01:59.143335 27448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:33895 every 8 connection(s)
I20260925 15:01:59.143703 27381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:59.144552 27449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:59.146796 27449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: Bootstrap starting.
I20260925 15:01:59.147382 27449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:59.147370 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27381
I20260925 15:01:59.147488 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:59.147720 8247 external_mini_cluster.cc:1428] Setting key 7fb62c422ebb6f2724de2e1b9604baef
I20260925 15:01:59.147720 27449 log.cc:824] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:59.148437 27449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: No bootstrap required, opened a new log
I20260925 15:01:59.150511 27449 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } }
I20260925 15:01:59.150645 27449 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:59.150674 27449 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e503d1750e439eb5f9e148104ae907, State: Initialized, Role: FOLLOWER
I20260925 15:01:59.150807 27449 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } }
I20260925 15:01:59.150875 27449 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:59.150906 27449 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:59.150951 27449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:59.151481 27449 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } }
I20260925 15:01:59.151595 27449 leader_election.cc:302] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47e503d1750e439eb5f9e148104ae907; no voters:
I20260925 15:01:59.151755 27449 leader_election.cc:288] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:59.151794 27454 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:59.151953 27454 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [term 1 LEADER]: Becoming Leader. State: Replica: 47e503d1750e439eb5f9e148104ae907, State: Running, Role: LEADER
I20260925 15:01:59.151988 27449 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:59.152051 27454 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } }
I20260925 15:01:59.152431 27455 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47e503d1750e439eb5f9e148104ae907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } } }
I20260925 15:01:59.152616 27456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e503d1750e439eb5f9e148104ae907. Latest consensus state: current_term: 1 leader_uuid: "47e503d1750e439eb5f9e148104ae907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47e503d1750e439eb5f9e148104ae907" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 33895 } } }
I20260925 15:01:59.152689 27455 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:59.152700 27456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907 [sys.catalog]: This master's current role is: LEADER
W20260925 15:01:59.153649 27469 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:01:59.153702 27469 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:01:59.153765 27465 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:59.153941 27465 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348518, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:59.155462 27465 catalog_manager.cc:1347] Generated new cluster ID: 31c1ba1212334a698eb9d2333b204712
I20260925 15:01:59.155506 27465 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:59.158828 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.148250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51722 (local address 127.8.13.254:33895)
0925 15:01:59.148420 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.148429 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.148456 (+ 27us) server_negotiation.cc:407] Connection header received
0925 15:01:59.148906 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.148913 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.148963 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.149094 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.149691 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.150570 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.151383 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.151956 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.152074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.154814 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.154818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.154818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:59.154830 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.156850 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.157241 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.157244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.157244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.157347 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.157561 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.157562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.157562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.157865 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.158015 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.158449 (+ 434us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.158598 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260925 15:01:59.165263 27465 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:59.165741 27465 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:59.168962 27465 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: Generated new TSK 0
I20260925 15:01:59.169111 27465 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.13.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
Entry for principal kudu/127.8.13.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab.
I20260925 15:01:59.198604 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:59.291158 27475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:59.291361 27475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:59.291384 27475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:59.292575 27475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:59.292618 27475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:59.292644 27475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:01:59.294073 27475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.193.keytab
--principal=kudu/127.8.13.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:59.294301 27475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:59.294517 27475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:59.299255 27483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.299310 27481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.299381 27475 server_base.cc:1034] running on GCE node
W20260925 15:01:59.299260 27480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.299649 27475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:59.299867 27475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:59.300993 27475 hybrid_clock.cc:648] HybridClock initialized: now 1790348519300980 us; error 30 us; skew 500 ppm
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:59.303321 27475 init.cc:377] Logged in from keytab as kudu/127.8.13.193@KRBTEST.COM (short username kudu)
I20260925 15:01:59.303812 27475 webserver.cc:460] Webserver started at http://127.8.13.193:46429/ using document root <none> and password file <none>
I20260925 15:01:59.304033 27475 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:59.304080 27475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:59.304199 27475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:59.305236 27475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "4e7d11c5977c4b28a619f70fe9c0f272"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "86b105f06a8431a94f1d5de49ae9cd4f"
server_key_iv: "7c03af1a8e6cc23b60eb896329e404ea"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.305553 27475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "4e7d11c5977c4b28a619f70fe9c0f272"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "86b105f06a8431a94f1d5de49ae9cd4f"
server_key_iv: "7c03af1a8e6cc23b60eb896329e404ea"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.306787 27475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:59.307477 27490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.307628 27475 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:59.307719 27475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "4e7d11c5977c4b28a619f70fe9c0f272"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "86b105f06a8431a94f1d5de49ae9cd4f"
server_key_iv: "7c03af1a8e6cc23b60eb896329e404ea"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.307777 27475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:59.334983 27475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:59.336823 27475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:59.336966 27475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:59.337184 27475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:59.337476 27475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:59.337512 27475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.337540 27475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:59.337556 27475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.343109 27475 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:46815
I20260925 15:01:59.343160 27603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:46815 every 8 connection(s)
I20260925 15:01:59.343467 27475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:01:59.343914 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27475
I20260925 15:01:59.344022 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:01:59.344261 8247 external_mini_cluster.cc:1428] Setting key ac9b2fda40ae1b83653777ceb0c3e765
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.193@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.194@KRBTEST.COM; defaulting to no policy
I20260925 15:01:59.353009 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.344218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43289 (local address 127.8.13.254:33895)
0925 15:01:59.344288 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.344290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.344490 (+ 200us) server_negotiation.cc:407] Connection header received
0925 15:01:59.345052 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.345053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.345064 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.345145 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.346079 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.346587 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.347596 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.347814 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.347872 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.350324 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.350326 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.350326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:59.350333 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.351651 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.352120 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.352121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.352122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.352175 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.352376 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.352376 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.352377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.352487 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.352535 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.352851 (+ 316us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.352939 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:01:59.353267 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.344189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33895 (local address 127.8.13.193:43289)
0925 15:01:59.344369 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.344412 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.344959 (+ 547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.345211 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.345214 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.345773 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:59.345970 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.345976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.346673 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.346674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.347492 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.347495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.347888 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.347889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.348021 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.348563 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:59.348571 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:59.350228 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:59.351732 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.351742 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.351745 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.352050 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.352227 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.352227 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.352228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.352331 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.352562 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:59.352573 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:59.352735 (+ 162us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.352830 (+ 95us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.352995 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260925 15:01:59.353863 27604 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33895
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.194@KRBTEST.COM" created.
I20260925 15:01:59.353964 27604 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:59.354143 27604 heartbeater.cc:511] Master 127.8.13.254:33895 requested a full tablet report, sending...
I20260925 15:01:59.354660 27413 ts_manager.cc:194] Registered new tserver with Master: 4e7d11c5977c4b28a619f70fe9c0f272 (127.8.13.193:46815)
I20260925 15:01:59.355262 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.193@KRBTEST.COM'} at 127.8.13.193:43289
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
Entry for principal kudu/127.8.13.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab.
I20260925 15:01:59.367266 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:59.447113 27609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:59.447324 27609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:59.447346 27609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:59.448480 27609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:59.448513 27609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:59.448537 27609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:01:59.449913 27609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.194.keytab
--principal=kudu/127.8.13.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:59.450098 27609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:59.450282 27609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260925 15:01:59.455173 27609 server_base.cc:1034] running on GCE node
W20260925 15:01:59.455117 27614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.455124 27617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.455111 27615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.455510 27609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:59.455780 27609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:59.456929 27609 hybrid_clock.cc:648] HybridClock initialized: now 1790348519456914 us; error 29 us; skew 500 ppm
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:59.459378 27609 init.cc:377] Logged in from keytab as kudu/127.8.13.194@KRBTEST.COM (short username kudu)
I20260925 15:01:59.459842 27609 webserver.cc:460] Webserver started at http://127.8.13.194:33599/ using document root <none> and password file <none>
I20260925 15:01:59.460050 27609 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:59.460098 27609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:59.460222 27609 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:59.461278 27609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "f780755580ca4233bb5cbf5ed02b6d30"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "66d18d5949cc785395eda6bca63f1c3c"
server_key_iv: "cd1a6a42f3f8857a52459e8a57d80cc5"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.461627 27609 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "f780755580ca4233bb5cbf5ed02b6d30"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "66d18d5949cc785395eda6bca63f1c3c"
server_key_iv: "cd1a6a42f3f8857a52459e8a57d80cc5"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.462836 27609 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:01:59.463475 27624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.463618 27609 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:01:59.463698 27609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "f780755580ca4233bb5cbf5ed02b6d30"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "66d18d5949cc785395eda6bca63f1c3c"
server_key_iv: "cd1a6a42f3f8857a52459e8a57d80cc5"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.463770 27609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:59.483258 27609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:59.485112 27609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:59.485251 27609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:59.485476 27609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:59.485795 27609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:59.485829 27609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.485857 27609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:59.485874 27609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.491220 27609 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:45343
I20260925 15:01:59.491268 27737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:45343 every 8 connection(s)
I20260925 15:01:59.491593 27609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:01:59.492065 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27609
I20260925 15:01:59.492184 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:01:59.492409 8247 external_mini_cluster.cc:1428] Setting key 4cfba77363e65279bfc78c968c153616
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.194@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.195@KRBTEST.COM; defaulting to no policy
I20260925 15:01:59.501216 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.492312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:52015 (local address 127.8.13.254:33895)
0925 15:01:59.492362 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.492364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.492607 (+ 243us) server_negotiation.cc:407] Connection header received
0925 15:01:59.493178 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.493179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.493189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.493222 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.494401 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.494883 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.496095 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.496390 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.496462 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.498520 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.498522 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.498522 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:59.498531 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.499987 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.500366 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.500367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.500368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.500429 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.500627 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.500627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.500628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.500770 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.500838 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.501056 (+ 218us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.501123 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:01:59.501390 27740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.492313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33895 (local address 127.8.13.194:52015)
0925 15:01:59.492477 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.492527 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.493088 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.493305 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.493310 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.494046 (+ 736us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:59.494319 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.494326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.494969 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.494970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.496021 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.496024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.496463 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.496463 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.496560 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.497108 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:59.497117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:59.498431 (+ 1314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:59.500066 (+ 1635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.500072 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.500074 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.500298 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.500485 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.500485 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.500486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.500576 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.500825 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:59.500829 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:59.500951 (+ 122us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.501030 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.501155 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260925 15:01:59.501825 27738 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33895
I20260925 15:01:59.501914 27738 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:59.502068 27738 heartbeater.cc:511] Master 127.8.13.254:33895 requested a full tablet report, sending...
I20260925 15:01:59.502408 27413 ts_manager.cc:194] Registered new tserver with Master: f780755580ca4233bb5cbf5ed02b6d30 (127.8.13.194:45343)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.195@KRBTEST.COM" created.
I20260925 15:01:59.502884 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.194@KRBTEST.COM'} at 127.8.13.194:52015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
Entry for principal kudu/127.8.13.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab.
I20260925 15:01:59.515936 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:01:59.593520 27743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:59.593739 27743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:59.593770 27743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:59.594934 27743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:59.594978 27743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:59.595013 27743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:01:59.596486 27743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.195.keytab
--principal=kudu/127.8.13.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:33895
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:59.596740 27743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:59.596951 27743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:59.601841 27751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.601835 27748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.601893 27743 server_base.cc:1034] running on GCE node
W20260925 15:01:59.601835 27749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.602259 27743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:59.602463 27743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:59.603595 27743 hybrid_clock.cc:648] HybridClock initialized: now 1790348519603579 us; error 30 us; skew 500 ppm
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:01:59.605998 27743 init.cc:377] Logged in from keytab as kudu/127.8.13.195@KRBTEST.COM (short username kudu)
I20260925 15:01:59.606443 27743 webserver.cc:460] Webserver started at http://127.8.13.195:35081/ using document root <none> and password file <none>
I20260925 15:01:59.606619 27743 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:59.606659 27743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:59.606750 27743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:59.607791 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "d3158862dd444d518fdc64aa2274a71a"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "f506b4d305e45d5c08c52e90bce4910b"
server_key_iv: "74af7151fb89e9aa67991c4604f52c0a"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.608083 27743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "d3158862dd444d518fdc64aa2274a71a"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "f506b4d305e45d5c08c52e90bce4910b"
server_key_iv: "74af7151fb89e9aa67991c4604f52c0a"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.609284 27743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:01:59.609915 27758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.610075 27743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:59.610149 27743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "d3158862dd444d518fdc64aa2274a71a"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "f506b4d305e45d5c08c52e90bce4910b"
server_key_iv: "74af7151fb89e9aa67991c4604f52c0a"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.610216 27743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:59.621968 27743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:59.624313 27743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:59.624451 27743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:59.624697 27743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:01:59.625032 27743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:01:59.625072 27743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.625099 27743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:01:59.625113 27743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.631210 27743 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:38563
I20260925 15:01:59.631278 27871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:38563 every 8 connection(s)
I20260925 15:01:59.631639 27743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
Sep 25 15:01:59 dist-test-slave-15hz krb5kdc[27368](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348519, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.195@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:01:59.640350 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27743
I20260925 15:01:59.640393 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.632384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:51623 (local address 127.8.13.254:33895)
0925 15:01:59.632463 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.632465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.632623 (+ 158us) server_negotiation.cc:407] Connection header received
0925 15:01:59.633128 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.633129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.633140 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.633202 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.634107 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.634613 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.635571 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.635795 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.635855 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.637865 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.637867 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.637867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:59.637874 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.639107 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.639497 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.639498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.639498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.639609 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.639811 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.639811 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.639812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.639914 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.639960 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.640252 (+ 292us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.640323 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:01:59.640478 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:01:59.640625 27874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.632318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:33895 (local address 127.8.13.195:51623)
0925 15:01:59.632498 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.632540 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.633021 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.633260 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.633264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.633830 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:01:59.634029 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.634034 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.634679 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.634679 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.635473 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.635475 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.635863 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.635864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.635965 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.636421 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:01:59.636429 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:01:59.637764 (+ 1335us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:01:59.639203 (+ 1439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.639209 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.639211 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.639435 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.639664 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:01:59.639664 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:01:59.639665 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:01:59.639764 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:01:59.639960 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:01:59.639964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:01:59.640085 (+ 121us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.640197 (+ 112us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.640365 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":105,"threads_started":1}
I20260925 15:01:59.640699 8247 external_mini_cluster.cc:1428] Setting key df2c9ef92fce777622ef04ba96cebb21
I20260925 15:01:59.641155 27872 heartbeater.cc:347] Connected to a master server at 127.8.13.254:33895
I20260925 15:01:59.641253 27872 heartbeater.cc:464] Registering TS with master...
I20260925 15:01:59.641450 27872 heartbeater.cc:511] Master 127.8.13.254:33895 requested a full tablet report, sending...
I20260925 15:01:59.641794 27413 ts_manager.cc:194] Registered new tserver with Master: d3158862dd444d518fdc64aa2274a71a (127.8.13.195:38563)
I20260925 15:01:59.642135 27413 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.13.195@KRBTEST.COM'} at 127.8.13.195:51623
I20260925 15:01:59.652554 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:01:59.659382 27453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.653880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51736 (local address 127.8.13.254:33895)
0925 15:01:59.653921 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.653923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.654022 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:01:59.654094 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.654095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.654106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.654150 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.654688 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.655185 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.655944 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.656145 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.656199 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.657276 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.657277 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.657277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:01:59.657285 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.658363 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.658654 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.658655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.658655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.658709 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:01:59.658859 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:01:59.658860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:01:59.658860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:01:59.659008 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.659093 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.659213 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.659291 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:01:59.660789 27413 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:01:59.662281 27413 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:01:59.669203 27881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:46815 (local address 127.0.0.1:33566)
0925 15:01:59.664898 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.664909 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.665155 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.665437 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.665438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.665439 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:59.665474 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.665476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.666710 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.666711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.668007 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.668008 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.668883 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.668883 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.668963 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.668979 (+ 16us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.669058 (+ 79us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.669113 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":30,"thread_start_us":63,"threads_started":1}
I20260925 15:01:59.669622 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33566 (local address 127.8.13.193:46815)
0925 15:01:59.664725 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.664729 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.664977 (+ 248us) server_negotiation.cc:407] Connection header received
0925 15:01:59.665258 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.665260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.665278 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.665330 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:59.665561 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.666619 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.668086 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.668812 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.668869 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.668915 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.669086 (+ 171us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.669130 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":86,"threads_started":1}
I20260925 15:01:59.669864 27880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:38563 (local address 127.0.0.1:57992)
0925 15:01:59.665020 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.665028 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.665083 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.665415 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.665417 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.665419 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:59.665474 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.665476 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.667726 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.667727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.668696 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.668699 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.669646 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.669647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.669703 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.669708 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.669757 (+ 49us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.669796 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":40,"threads_started":1}
I20260925 15:01:59.670094 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57992 (local address 127.8.13.195:38563)
0925 15:01:59.664876 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.664880 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.665076 (+ 196us) server_negotiation.cc:407] Connection header received
0925 15:01:59.665229 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.665232 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.665250 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.665330 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:59.666712 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.667635 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.668790 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.669488 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.669554 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.669612 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.669972 (+ 360us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.670022 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":88,"threads_started":1}
I20260925 15:01:59.670411 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45343 (local address 127.0.0.1:46172)
0925 15:01:59.664772 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.664810 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.664876 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.665207 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.665210 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.665213 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:59.665281 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.665283 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.666570 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.666571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.668003 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.668006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.670165 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.670166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.670238 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.670242 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.670294 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.670334 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"mutex_wait_us":3,"spinlock_wait_cycles":768,"thread_start_us":64,"threads_started":1}
I20260925 15:01:59.670372 27538 tablet_service.cc:1467] Processing CreateTablet for tablet 091dac628b7b411aa34aaae652e233e0 (DEFAULT_TABLE table=test-table [id=7c54be3f914b4a28aaaf5969a4d5a857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:59.670646 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.664732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46172 (local address 127.8.13.194:45343)
0925 15:01:59.664981 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.664984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.664993 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:01:59.665031 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.665033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.665049 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.665119 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:59.665357 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.666485 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.668947 (+ 2462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.669629 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.669697 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.669754 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.670515 (+ 761us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.670563 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215,"thread_start_us":72,"threads_started":1}
I20260925 15:01:59.670743 27538 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091dac628b7b411aa34aaae652e233e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:59.670730 27806 tablet_service.cc:1467] Processing CreateTablet for tablet 091dac628b7b411aa34aaae652e233e0 (DEFAULT_TABLE table=test-table [id=7c54be3f914b4a28aaaf5969a4d5a857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:59.671034 27806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091dac628b7b411aa34aaae652e233e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:59.674064 27887 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: Bootstrap starting.
I20260925 15:01:59.674645 27887 tablet_bootstrap.cc:654] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:59.674612 27672 tablet_service.cc:1467] Processing CreateTablet for tablet 091dac628b7b411aa34aaae652e233e0 (DEFAULT_TABLE table=test-table [id=7c54be3f914b4a28aaaf5969a4d5a857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:01:59.674901 27672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 091dac628b7b411aa34aaae652e233e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:59.675119 27887 log.cc:824] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:59.675448 27886 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Bootstrap starting.
I20260925 15:01:59.675882 27887 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: No bootstrap required, opened a new log
I20260925 15:01:59.675954 27887 ts_tablet_manager.cc:1397] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:01:59.676311 27886 tablet_bootstrap.cc:654] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:59.676744 27886 log.cc:824] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:59.677372 27886 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: No bootstrap required, opened a new log
I20260925 15:01:59.677429 27886 ts_tablet_manager.cc:1397] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:01:59.677851 27887 raft_consensus.cc:348] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.678038 27887 raft_consensus.cc:374] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:59.678084 27887 raft_consensus.cc:729] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e7d11c5977c4b28a619f70fe9c0f272, State: Initialized, Role: FOLLOWER
I20260925 15:01:59.678217 27890 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Bootstrap starting.
I20260925 15:01:59.678228 27887 consensus_queue.cc:260] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.678488 27887 ts_tablet_manager.cc:1428] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260925 15:01:59.678607 27604 heartbeater.cc:503] Master 127.8.13.254:33895 was elected leader, sending a full tablet report...
I20260925 15:01:59.678849 27890 tablet_bootstrap.cc:654] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:59.679136 27886 raft_consensus.cc:348] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.679312 27886 raft_consensus.cc:374] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:59.679351 27886 raft_consensus.cc:729] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3158862dd444d518fdc64aa2274a71a, State: Initialized, Role: FOLLOWER
I20260925 15:01:59.679359 27890 log.cc:824] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:59.679464 27886 consensus_queue.cc:260] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.679780 27872 heartbeater.cc:503] Master 127.8.13.254:33895 was elected leader, sending a full tablet report...
I20260925 15:01:59.679818 27886 ts_tablet_manager.cc:1428] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260925 15:01:59.680160 27890 tablet_bootstrap.cc:492] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: No bootstrap required, opened a new log
I20260925 15:01:59.680219 27890 ts_tablet_manager.cc:1397] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:01:59.681587 27890 raft_consensus.cc:348] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.681732 27890 raft_consensus.cc:374] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:59.681766 27890 raft_consensus.cc:729] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f780755580ca4233bb5cbf5ed02b6d30, State: Initialized, Role: FOLLOWER
I20260925 15:01:59.681871 27890 consensus_queue.cc:260] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.682117 27890 ts_tablet_manager.cc:1428] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260925 15:01:59.682156 27738 heartbeater.cc:503] Master 127.8.13.254:33895 was elected leader, sending a full tablet report...
I20260925 15:01:59.735704 27891 raft_consensus.cc:482] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:01:59.735844 27891 raft_consensus.cc:504] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.736234 27891 leader_election.cc:288] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f780755580ca4233bb5cbf5ed02b6d30 (127.8.13.194:45343), d3158862dd444d518fdc64aa2274a71a (127.8.13.195:38563)
I20260925 15:01:59.739341 27884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.736488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:56135 (local address 127.8.13.195:38563)
0925 15:01:59.736530 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.736532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.736738 (+ 206us) server_negotiation.cc:407] Connection header received
0925 15:01:59.736776 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.736777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.736788 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.736843 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:59.737032 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.737566 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.738597 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.738977 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.739038 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.739049 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.739228 (+ 179us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.739270 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:01:59.739346 27885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.736558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:43329 (local address 127.8.13.194:45343)
0925 15:01:59.736601 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.736603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.736611 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:01:59.736642 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.736643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.736655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.736747 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:01:59.737028 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.737582 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.738587 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.738981 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.739047 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.739057 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.739214 (+ 157us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.739265 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:01:59.739365 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.736438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:45343 (local address 127.8.13.193:43329)
0925 15:01:59.736499 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.736512 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.736557 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.736897 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.736898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.736900 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:59.736964 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.736965 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.737659 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.737660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.738494 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.738497 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.739064 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.739065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.739148 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.739151 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.739197 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.739274 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":49}
I20260925 15:01:59.739373 27895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.736438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:38563 (local address 127.8.13.193:56135)
0925 15:01:59.736657 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.736667 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:01:59.736728 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:01:59.736865 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:01:59.736866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:01:59.736868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:01:59.736946 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.736948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.737652 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.737653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.738492 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:01:59.738494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.739063 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.739064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:01:59.739151 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.739155 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:01:59.739200 (+ 45us) client_negotiation.cc:240] Negotiation successful
0925 15:01:59.739251 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":21,"thread_start_us":117,"threads_started":1}
I20260925 15:01:59.739775 27826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091dac628b7b411aa34aaae652e233e0" candidate_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3158862dd444d518fdc64aa2274a71a" is_pre_election: true
I20260925 15:01:59.739763 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091dac628b7b411aa34aaae652e233e0" candidate_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f780755580ca4233bb5cbf5ed02b6d30" is_pre_election: true
I20260925 15:01:59.739953 27692 raft_consensus.cc:2454] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7d11c5977c4b28a619f70fe9c0f272 in term 0.
I20260925 15:01:59.739953 27826 raft_consensus.cc:2454] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e7d11c5977c4b28a619f70fe9c0f272 in term 0.
I20260925 15:01:59.740209 27491 leader_election.cc:302] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7d11c5977c4b28a619f70fe9c0f272, d3158862dd444d518fdc64aa2274a71a; no voters:
I20260925 15:01:59.740355 27891 raft_consensus.cc:2781] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:01:59.740411 27891 raft_consensus.cc:482] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:01:59.740427 27891 raft_consensus.cc:3037] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:59.741101 27891 raft_consensus.cc:504] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.741235 27891 leader_election.cc:288] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [CANDIDATE]: Term 1 election: Requested vote from peers f780755580ca4233bb5cbf5ed02b6d30 (127.8.13.194:45343), d3158862dd444d518fdc64aa2274a71a (127.8.13.195:38563)
I20260925 15:01:59.741401 27826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091dac628b7b411aa34aaae652e233e0" candidate_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3158862dd444d518fdc64aa2274a71a"
I20260925 15:01:59.741401 27692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "091dac628b7b411aa34aaae652e233e0" candidate_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f780755580ca4233bb5cbf5ed02b6d30"
I20260925 15:01:59.741492 27826 raft_consensus.cc:3037] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:59.741492 27692 raft_consensus.cc:3037] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:01:59.742120 27739 tablet.cc:2369] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:01:59.742422 27826 raft_consensus.cc:2454] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7d11c5977c4b28a619f70fe9c0f272 in term 1.
I20260925 15:01:59.742422 27692 raft_consensus.cc:2454] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e7d11c5977c4b28a619f70fe9c0f272 in term 1.
I20260925 15:01:59.742579 27491 leader_election.cc:302] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e7d11c5977c4b28a619f70fe9c0f272, d3158862dd444d518fdc64aa2274a71a; no voters:
I20260925 15:01:59.742689 27891 raft_consensus.cc:2781] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:59.742863 27891 raft_consensus.cc:686] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [term 1 LEADER]: Becoming Leader. State: Replica: 4e7d11c5977c4b28a619f70fe9c0f272, State: Running, Role: LEADER
I20260925 15:01:59.742961 27891 consensus_queue.cc:237] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } }
I20260925 15:01:59.743840 27413 catalog_manager.cc:5495] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e7d11c5977c4b28a619f70fe9c0f272 (127.8.13.193). New cstate: current_term: 1 leader_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e7d11c5977c4b28a619f70fe9c0f272" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 46815 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 } health_report { overall_health: UNKNOWN } } }
I20260925 15:01:59.759915 27882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.757354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.8.13.193:46815)
0925 15:01:59.757395 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.757397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.757405 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:01:59.757472 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.757473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.757492 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.757549 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:01:59.757661 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.758326 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.759188 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.759384 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.759433 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.759616 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:01:59.759733 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:01:59.759791 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.759834 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.759853 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:01:59.761655 27826 raft_consensus.cc:1261] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 1 FOLLOWER]: Refusing update from remote peer 4e7d11c5977c4b28a619f70fe9c0f272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:59.761971 27692 raft_consensus.cc:1261] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 1 FOLLOWER]: Refusing update from remote peer 4e7d11c5977c4b28a619f70fe9c0f272: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:01:59.761991 27891 consensus_queue.cc:1035] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3158862dd444d518fdc64aa2274a71a" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 38563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:59.762245 27891 consensus_queue.cc:1035] T 091dac628b7b411aa34aaae652e233e0 P 4e7d11c5977c4b28a619f70fe9c0f272 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f780755580ca4233bb5cbf5ed02b6d30" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 45343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:01:59.764945 27901 mvcc.cc:205] Tried to move back new op lower bound from 7333267536941293568 to 7333267536867872768. Current Snapshot: MvccSnapshot[applied={T|T < 7333267536941293568}]
I20260925 15:01:59.766567 27413 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:59.766647 27413 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51736:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:01:59.767431 27413 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 47e503d1750e439eb5f9e148104ae907: Sending DeleteTablet for 3 replicas of tablet 091dac628b7b411aa34aaae652e233e0
I20260925 15:01:59.767789 27672 tablet_service.cc:1514] Processing DeleteTablet for tablet 091dac628b7b411aa34aaae652e233e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:46172
I20260925 15:01:59.767830 27538 tablet_service.cc:1514] Processing DeleteTablet for tablet 091dac628b7b411aa34aaae652e233e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:33566
I20260925 15:01:59.767838 27806 tablet_service.cc:1514] Processing DeleteTablet for tablet 091dac628b7b411aa34aaae652e233e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:01:59 UTC) from {username='kudu', principal='kudu/127.8.13.254@KRBTEST.COM'} at 127.0.0.1:57992
I20260925 15:01:59.768081 27909 tablet_replica.cc:330] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: stopping tablet replica
I20260925 15:01:59.768182 27909 raft_consensus.cc:2229] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:59.768208 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27475
I20260925 15:01:59.769053 27910 tablet_replica.cc:330] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: stopping tablet replica
I20260925 15:01:59.769099 27909 raft_consensus.cc:2258] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:59.769127 27910 raft_consensus.cc:2229] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:01:59.769196 27910 raft_consensus.cc:2258] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:01:59.769567 27910 ts_tablet_manager.cc:1905] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:59.769586 27909 ts_tablet_manager.cc:1905] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:01:59.770844 27910 ts_tablet_manager.cc:1918] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:59.770889 27910 log.cc:1196] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/091dac628b7b411aa34aaae652e233e0
I20260925 15:01:59.771034 27909 ts_tablet_manager.cc:1918] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:01:59.771090 27909 log.cc:1196] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/091dac628b7b411aa34aaae652e233e0
I20260925 15:01:59.771091 27910 ts_tablet_manager.cc:1939] T 091dac628b7b411aa34aaae652e233e0 P d3158862dd444d518fdc64aa2274a71a: Deleting consensus metadata
I20260925 15:01:59.771310 27909 ts_tablet_manager.cc:1939] T 091dac628b7b411aa34aaae652e233e0 P f780755580ca4233bb5cbf5ed02b6d30: Deleting consensus metadata
I20260925 15:01:59.771416 27397 catalog_manager.cc:4841] TS d3158862dd444d518fdc64aa2274a71a (127.8.13.195:38563): tablet 091dac628b7b411aa34aaae652e233e0 (table test-table [id=7c54be3f914b4a28aaaf5969a4d5a857]) successfully deleted
I20260925 15:01:59.771762 27399 catalog_manager.cc:4841] TS f780755580ca4233bb5cbf5ed02b6d30 (127.8.13.194:45343): tablet 091dac628b7b411aa34aaae652e233e0 (table test-table [id=7c54be3f914b4a28aaaf5969a4d5a857]) successfully deleted
W20260925 15:01:59.773077 27397 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:46815 (error 108)
I20260925 15:01:59.773505 27883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.773400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:46815 (local address 127.0.0.1:33580)
0925 15:01:59.773444 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:01:59.773477 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:46815: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:01:59.773607 27397 catalog_manager.cc:4584] TS 4e7d11c5977c4b28a619f70fe9c0f272 (127.8.13.193:46815): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 091dac628b7b411aa34aaae652e233e0: Network error: Client connection negotiation failed: client connection to 127.8.13.193:46815: connect: Connection refused (error 111)
I20260925 15:01:59.773851 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27609
I20260925 15:01:59.778432 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27743
I20260925 15:01:59.782423 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27381
2026-09-25T15:01:59Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1836 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-25T15:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:01:59Z Disabled control of system clock
I20260925 15:01:59.807905 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:41169
--webserver_interface=127.8.13.254
--webserver_port=0
--builtin_ntp_servers=127.8.13.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:41169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:01:59.888509 27916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:01:59.888736 27916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:01:59.888754 27916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:01:59.889850 27916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:01:59.889882 27916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:01:59.889894 27916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:01:59.889904 27916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:01:59.889923 27916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:01:59.891182 27916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:41169
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:41169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:01:59.891377 27916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:01:59.891638 27916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:01:59.896042 27924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:01:59.896081 27921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.896241 27916 server_base.cc:1034] running on GCE node
W20260925 15:01:59.896288 27922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:01:59.896534 27916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:01:59.896790 27916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:01:59.897936 27916 hybrid_clock.cc:648] HybridClock initialized: now 1790348519897917 us; error 33 us; skew 500 ppm
I20260925 15:01:59.899307 27916 webserver.cc:460] Webserver started at http://127.8.13.254:32829/ using document root <none> and password file <none>
I20260925 15:01:59.899582 27916 fs_manager.cc:359] Metadata directory not provided
I20260925 15:01:59.899653 27916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:01:59.899787 27916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:01:59.900866 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "d8fcef4a99c74971b735e0130b015473"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "584ca4715fec919c3ccb6922060ad160"
server_key_iv: "23b245319ecf0123ccc3f20e732fb02b"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.901212 27916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "d8fcef4a99c74971b735e0130b015473"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "584ca4715fec919c3ccb6922060ad160"
server_key_iv: "23b245319ecf0123ccc3f20e732fb02b"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.902503 27916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:01:59.903198 27930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:01:59.903363 27916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:01:59.903446 27916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "d8fcef4a99c74971b735e0130b015473"
format_stamp: "Formatted at 2026-09-25 15:01:59 on dist-test-slave-15hz"
server_key: "584ca4715fec919c3ccb6922060ad160"
server_key_iv: "23b245319ecf0123ccc3f20e732fb02b"
server_key_version: "encryptionkey@0"
I20260925 15:01:59.903515 27916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:01:59.923475 27916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:01:59.923853 27916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:01:59.923986 27916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:01:59.927709 27916 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:41169
I20260925 15:01:59.927750 27982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:41169 every 8 connection(s)
I20260925 15:01:59.928077 27916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:01:59.928987 27983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:01:59.931233 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: Bootstrap starting.
I20260925 15:01:59.931847 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 27916
I20260925 15:01:59.931881 27983 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: Neither blocks nor log segments found. Creating new log.
I20260925 15:01:59.931955 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:01:59.932171 8247 external_mini_cluster.cc:1428] Setting key 72668e5b75c6bbb616e143082c20fb4a
I20260925 15:01:59.932186 27983 log.cc:824] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: Log is configured to *not* fsync() on all Append() calls
I20260925 15:01:59.932909 27983 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: No bootstrap required, opened a new log
I20260925 15:01:59.935102 27983 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } }
I20260925 15:01:59.935257 27983 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:01:59.935297 27983 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fcef4a99c74971b735e0130b015473, State: Initialized, Role: FOLLOWER
I20260925 15:01:59.935467 27983 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } }
I20260925 15:01:59.935576 27983 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:01:59.935613 27983 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:01:59.935657 27983 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:01:59.936362 27983 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } }
I20260925 15:01:59.936463 27983 leader_election.cc:302] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8fcef4a99c74971b735e0130b015473; no voters:
I20260925 15:01:59.936614 27983 leader_election.cc:288] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:01:59.936743 27988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:01:59.936882 27988 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [term 1 LEADER]: Becoming Leader. State: Replica: d8fcef4a99c74971b735e0130b015473, State: Running, Role: LEADER
I20260925 15:01:59.936904 27983 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:01:59.936964 27988 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } }
I20260925 15:01:59.937263 27988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8fcef4a99c74971b735e0130b015473. Latest consensus state: current_term: 1 leader_uuid: "d8fcef4a99c74971b735e0130b015473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } } }
I20260925 15:01:59.937333 27988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:59.937599 27989 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8fcef4a99c74971b735e0130b015473" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fcef4a99c74971b735e0130b015473" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 41169 } } }
I20260925 15:01:59.937681 27989 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473 [sys.catalog]: This master's current role is: LEADER
I20260925 15:01:59.937824 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:01:59.932653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.8.13.254:41169)
0925 15:01:59.932843 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0925 15:01:59.932852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:01:59.932886 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:01:59.933338 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:01:59.933344 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:01:59.933404 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:01:59.933524 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:01:59.933748 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.934614 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.935450 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:01:59.936093 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:01:59.936235 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:01:59.936902 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:01:59.936906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:01:59.936906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:01:59.937105 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:01:59.937150 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:01:59.937161 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:01:59.937212 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:01:59.937302 (+ 90us) server_negotiation.cc:299] Negotiation successful
0925 15:01:59.937384 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260925 15:01:59.937867 27994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:01:59.938342 27994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:01:59.939883 27994 catalog_manager.cc:1347] Generated new cluster ID: 905dc13ee023474896b82c5832299c24
I20260925 15:01:59.939941 27994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:01:59.964206 27994 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:01:59.964936 27994 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:01:59.975675 27994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: Generated new TSK 0
I20260925 15:01:59.975912 27994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:01:59.981356 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.193:0
--local_ip_for_outbound_sockets=127.8.13.193
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--builtin_ntp_servers=127.8.13.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260925 15:02:00.062356 28007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:00.062572 28007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:00.062595 28007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:00.063802 28007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:00.063839 28007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:00.063864 28007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.193
I20260925 15:02:00.065218 28007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.13.193
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28007
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.193
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:00.065450 28007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:00.065729 28007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:00.070655 28012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:00.070659 28013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.070745 28007 server_base.cc:1034] running on GCE node
W20260925 15:02:00.070655 28015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.071096 28007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:00.071334 28007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:00.072474 28007 hybrid_clock.cc:648] HybridClock initialized: now 1790348520072456 us; error 29 us; skew 500 ppm
I20260925 15:02:00.073762 28007 webserver.cc:460] Webserver started at http://127.8.13.193:37889/ using document root <none> and password file <none>
I20260925 15:02:00.074002 28007 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:00.074050 28007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:00.074184 28007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:00.075295 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data/instance:
uuid: "6b322aeffc8d4018be42f3b3ee239b0d"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "79fdd562eaec0d211a579041b9f7b79d"
server_key_iv: "db7ef7b5ab07c48ffafb6bfcec86040e"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.075680 28007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance:
uuid: "6b322aeffc8d4018be42f3b3ee239b0d"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "79fdd562eaec0d211a579041b9f7b79d"
server_key_iv: "db7ef7b5ab07c48ffafb6bfcec86040e"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.076944 28007 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:02:00.077661 28021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.077824 28007 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:02:00.077903 28007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal
uuid: "6b322aeffc8d4018be42f3b3ee239b0d"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "79fdd562eaec0d211a579041b9f7b79d"
server_key_iv: "db7ef7b5ab07c48ffafb6bfcec86040e"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.077968 28007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:00.087185 28007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:00.087457 28007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:00.087604 28007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:00.087827 28007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:00.088127 28007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:00.088158 28007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.088186 28007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:00.088207 28007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.094266 28007 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.193:41891
I20260925 15:02:00.094328 28134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.193:41891 every 8 connection(s)
I20260925 15:02:00.094630 28007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/data/info.pb
I20260925 15:02:00.096030 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28007
I20260925 15:02:00.096123 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-0/wal/instance
I20260925 15:02:00.096323 8247 external_mini_cluster.cc:1428] Setting key 53d7ff48c0c6270b307dba6b93dd9db7
I20260925 15:02:00.097222 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.194:0
--local_ip_for_outbound_sockets=127.8.13.194
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--builtin_ntp_servers=127.8.13.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:02:00.100165 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.095301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.193:42529 (local address 127.8.13.254:41169)
0925 15:02:00.095358 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.095359 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.095589 (+ 230us) server_negotiation.cc:407] Connection header received
0925 15:02:00.096139 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.096140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.096153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.096210 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:00.096934 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.097503 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.098715 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.098911 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.098967 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.099914 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:00.099916 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:00.099916 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:00.099921 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:00.099936 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:00.099940 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:00.099981 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.100054 (+ 73us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.100083 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260925 15:02:00.100389 28137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.095292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:41169 (local address 127.8.13.193:42529)
0925 15:02:00.095463 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.095499 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.096037 (+ 538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.096283 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.096287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.096297 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:00.096847 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.096853 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.097597 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.097599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.098619 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.098621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.098993 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.098994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.099155 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.099775 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:02:00.099776 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:00.099807 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:02:00.099830 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:00.099989 (+ 159us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:00.099994 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:00.099997 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.100075 (+ 78us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.100182 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":68,"threads_started":1}
I20260925 15:02:00.100863 28135 heartbeater.cc:347] Connected to a master server at 127.8.13.254:41169
I20260925 15:02:00.100950 28135 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:00.101131 28135 heartbeater.cc:511] Master 127.8.13.254:41169 requested a full tablet report, sending...
I20260925 15:02:00.101634 27947 ts_manager.cc:194] Registered new tserver with Master: 6b322aeffc8d4018be42f3b3ee239b0d (127.8.13.193:41891)
I20260925 15:02:00.102218 27947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.193:42529
W20260925 15:02:00.177886 28138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:00.178099 28138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:00.178129 28138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:00.179265 28138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:00.179302 28138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:00.179337 28138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.194
I20260925 15:02:00.180719 28138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.13.194
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.194
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:00.180945 28138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:00.181214 28138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:00.186115 28143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.186167 28138 server_base.cc:1034] running on GCE node
W20260925 15:02:00.186129 28144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:00.186110 28146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.186517 28138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:00.186753 28138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:00.187883 28138 hybrid_clock.cc:648] HybridClock initialized: now 1790348520187875 us; error 32 us; skew 500 ppm
I20260925 15:02:00.189081 28138 webserver.cc:460] Webserver started at http://127.8.13.194:42985/ using document root <none> and password file <none>
I20260925 15:02:00.189291 28138 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:00.189338 28138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:00.189488 28138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:00.190529 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data/instance:
uuid: "b40207091924470087669119d5c555bd"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "ad3703178692b78b0e702f7af64af3a2"
server_key_iv: "5d234080cf9622c13a39757dfc6397c0"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.190864 28138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance:
uuid: "b40207091924470087669119d5c555bd"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "ad3703178692b78b0e702f7af64af3a2"
server_key_iv: "5d234080cf9622c13a39757dfc6397c0"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.192109 28138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:00.192811 28152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.192968 28138 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:00.193045 28138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal
uuid: "b40207091924470087669119d5c555bd"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "ad3703178692b78b0e702f7af64af3a2"
server_key_iv: "5d234080cf9622c13a39757dfc6397c0"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.193109 28138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:00.204348 28138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:00.204648 28138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:00.204770 28138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:00.204981 28138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:00.205277 28138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:00.205309 28138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.205339 28138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:00.205363 28138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.210855 28138 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.194:44677
I20260925 15:02:00.210922 28265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.194:44677 every 8 connection(s)
I20260925 15:02:00.211236 28138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/data/info.pb
I20260925 15:02:00.212210 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28138
I20260925 15:02:00.212311 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal/instance
I20260925 15:02:00.212487 8247 external_mini_cluster.cc:1428] Setting key 871d293dacb89da1245a0550dc60d988
I20260925 15:02:00.213419 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.13.195:0
--local_ip_for_outbound_sockets=127.8.13.195
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--builtin_ntp_servers=127.8.13.212:33531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260925 15:02:00.217435 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.211968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:50615 (local address 127.8.13.254:41169)
0925 15:02:00.212013 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.212015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.212267 (+ 252us) server_negotiation.cc:407] Connection header received
0925 15:02:00.213046 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.213046 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.213054 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.213124 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:00.213858 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.214534 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.216011 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.216201 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.216256 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.217190 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:00.217191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:00.217191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:00.217196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:00.217212 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:00.217216 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:00.217257 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.217348 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.217372 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:02:00.217727 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.211950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:41169 (local address 127.8.13.194:50615)
0925 15:02:00.212132 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.212174 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.212750 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.213198 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.213202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.213210 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:00.213765 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.213771 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.214667 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.214668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.215921 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.215925 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.216298 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.216299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.216458 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.217029 (+ 571us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:02:00.217029 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:00.217073 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:02:00.217106 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:00.217273 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:00.217278 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:00.217281 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.217354 (+ 73us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.217473 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
I20260925 15:02:00.218216 28266 heartbeater.cc:347] Connected to a master server at 127.8.13.254:41169
I20260925 15:02:00.218307 28266 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:00.218482 28266 heartbeater.cc:511] Master 127.8.13.254:41169 requested a full tablet report, sending...
I20260925 15:02:00.218842 27947 ts_manager.cc:194] Registered new tserver with Master: b40207091924470087669119d5c555bd (127.8.13.194:44677)
I20260925 15:02:00.219343 27947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.194:50615
W20260925 15:02:00.293329 28269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:00.293527 28269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:00.293550 28269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:00.294667 28269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:00.294699 28269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:00.294723 28269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.13.195
I20260925 15:02:00.296077 28269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.13.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.13.195
--webserver_port=0
--tserver_master_addrs=127.8.13.254:41169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.13.195
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:00.296271 28269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:00.296516 28269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:00.301369 28275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:00.301364 28277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.301489 28269 server_base.cc:1034] running on GCE node
W20260925 15:02:00.301431 28274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:00.301810 28269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:00.302042 28269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:00.303202 28269 hybrid_clock.cc:648] HybridClock initialized: now 1790348520303187 us; error 31 us; skew 500 ppm
I20260925 15:02:00.304533 28269 webserver.cc:460] Webserver started at http://127.8.13.195:40813/ using document root <none> and password file <none>
I20260925 15:02:00.304778 28269 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:00.304831 28269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:00.304960 28269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:00.306020 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data/instance:
uuid: "31658c4cffb349f990ba7d81e36fca56"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "7618008e8daf52cbe16cc7b10397d551"
server_key_iv: "ca3e5910c4a84017a241d97ca6bba6ac"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.306360 28269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance:
uuid: "31658c4cffb349f990ba7d81e36fca56"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "7618008e8daf52cbe16cc7b10397d551"
server_key_iv: "ca3e5910c4a84017a241d97ca6bba6ac"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.307675 28269 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:00.308553 28283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.308765 28269 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:00.308854 28269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal
uuid: "31658c4cffb349f990ba7d81e36fca56"
format_stamp: "Formatted at 2026-09-25 15:02:00 on dist-test-slave-15hz"
server_key: "7618008e8daf52cbe16cc7b10397d551"
server_key_iv: "ca3e5910c4a84017a241d97ca6bba6ac"
server_key_version: "encryptionkey@0"
I20260925 15:02:00.308925 28269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:00.317206 28269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:00.317516 28269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:00.317653 28269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:00.317866 28269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260925 15:02:00.318215 28269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260925 15:02:00.318248 28269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.318275 28269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260925 15:02:00.318292 28269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:00.323849 28269 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.195:43563
I20260925 15:02:00.323889 28396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.195:43563 every 8 connection(s)
I20260925 15:02:00.324220 28269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/data/info.pb
I20260925 15:02:00.329100 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28269
I20260925 15:02:00.329213 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal/instance
I20260925 15:02:00.329361 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.324953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.195:32855 (local address 127.8.13.254:41169)
0925 15:02:00.325016 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.325018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.325228 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:02:00.325740 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.325742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.325755 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.325809 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:00.326514 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.327007 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.328040 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.328232 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.328288 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.329078 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:00.329079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:00.329079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:00.329085 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:00.329101 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:00.329104 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:00.329147 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.329245 (+ 98us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.329275 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:02:00.329423 8247 external_mini_cluster.cc:1428] Setting key 5c322aa4a78578e1cb46ed9b29bdff7b
I20260925 15:02:00.329569 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.324922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:41169 (local address 127.8.13.195:32855)
0925 15:02:00.325103 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.325140 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.325637 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.325879 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.325882 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.325891 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:00.326425 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.326429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.327089 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.327090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.327957 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.327960 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.328302 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.328303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.328432 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.328944 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0925 15:02:00.328945 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:00.328976 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0925 15:02:00.329000 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:00.329152 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:00.329179 (+ 27us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:00.329182 (+ 3us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.329242 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.329354 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":76,"threads_started":1}
I20260925 15:02:00.330031 28397 heartbeater.cc:347] Connected to a master server at 127.8.13.254:41169
I20260925 15:02:00.330135 28397 heartbeater.cc:464] Registering TS with master...
I20260925 15:02:00.330339 28397 heartbeater.cc:511] Master 127.8.13.254:41169 requested a full tablet report, sending...
I20260925 15:02:00.330662 27947 ts_manager.cc:194] Registered new tserver with Master: 31658c4cffb349f990ba7d81e36fca56 (127.8.13.195:43563)
I20260925 15:02:00.331022 27947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.13.195:32855
I20260925 15:02:00.341331 8247 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260925 15:02:00.345791 27987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.342733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45102 (local address 127.8.13.254:41169)
0925 15:02:00.342806 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.342808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.342872 (+ 64us) server_negotiation.cc:407] Connection header received
0925 15:02:00.342958 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.342959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.342969 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.343008 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:00.343208 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.343818 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.344577 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.344759 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.344814 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.345543 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:00.345545 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:00.345545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:00.345551 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:00.345570 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:00.345574 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:00.345614 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.345673 (+ 59us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.345700 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260925 15:02:00.347358 27947 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260925 15:02:00.348978 27947 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260925 15:02:00.355357 28406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.194:44677 (local address 127.0.0.1:38412)
0925 15:02:00.351707 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.351718 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.351778 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.351931 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.351932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.351934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:00.351982 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.351984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.352942 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.352943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.354177 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.354179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.355042 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.355043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.355120 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.355135 (+ 15us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.355209 (+ 74us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.355262 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"mutex_wait_us":238,"thread_start_us":71,"threads_started":1}
I20260925 15:02:00.355471 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52778 (local address 127.8.13.193:41891)
0925 15:02:00.351684 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.351689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.351698 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:02:00.351735 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.351737 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.351750 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.351843 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:00.352038 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.353215 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.354406 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.355085 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.355193 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.355283 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.355343 (+ 60us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.355382 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":94,"threads_started":1}
I20260925 15:02:00.355772 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38412 (local address 127.8.13.194:44677)
0925 15:02:00.351652 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.351657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.351793 (+ 136us) server_negotiation.cc:407] Connection header received
0925 15:02:00.351839 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.351841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.351856 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.351922 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:00.352098 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.352865 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.354246 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.354959 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.355036 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.355135 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.355650 (+ 515us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.355693 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":72,"threads_started":1}
I20260925 15:02:00.355471 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41891 (local address 127.0.0.1:52778)
0925 15:02:00.351403 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.351446 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.351512 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.351856 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.351859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.351861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:00.351950 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.351952 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.353301 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.353302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.354331 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.354332 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.355202 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.355204 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.355271 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.355276 (+ 5us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.355329 (+ 53us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.355373 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260925 15:02:00.356426 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43563 (local address 127.0.0.1:53942)
0925 15:02:00.351594 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.351603 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.351644 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.351890 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.351890 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.351892 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:00.351950 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.351953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.354280 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.354280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.354981 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.354983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.356218 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.356219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.356279 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.356283 (+ 4us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.356327 (+ 44us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.356364 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":24,"thread_start_us":93,"threads_started":1}
I20260925 15:02:00.356710 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.351456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53942 (local address 127.8.13.195:43563)
0925 15:02:00.351652 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.351657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.351689 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:02:00.351733 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.351735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.351751 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.351820 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:00.352032 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.353218 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.355070 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.356036 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.356104 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.356183 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.356575 (+ 392us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.356618 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":93,"threads_started":1}
I20260925 15:02:00.356630 28200 tablet_service.cc:1467] Processing CreateTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 (DEFAULT_TABLE table=test-table [id=ebb69fc91cf64df8b0adb18b0cb0d6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:00.356685 28068 tablet_service.cc:1467] Processing CreateTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 (DEFAULT_TABLE table=test-table [id=ebb69fc91cf64df8b0adb18b0cb0d6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:00.357009 28200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46243cc3702549d0a4690d9920e2e0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:00.357018 28068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46243cc3702549d0a4690d9920e2e0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:00.357306 28331 tablet_service.cc:1467] Processing CreateTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 (DEFAULT_TABLE table=test-table [id=ebb69fc91cf64df8b0adb18b0cb0d6c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260925 15:02:00.357620 28331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46243cc3702549d0a4690d9920e2e0a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:00.359642 28411 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Bootstrap starting.
I20260925 15:02:00.360206 28411 tablet_bootstrap.cc:654] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:00.360486 28411 log.cc:824] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:00.360515 28412 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: Bootstrap starting.
I20260925 15:02:00.361078 28412 tablet_bootstrap.cc:654] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:00.361152 28411 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: No bootstrap required, opened a new log
I20260925 15:02:00.361229 28411 ts_tablet_manager.cc:1397] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260925 15:02:00.361338 28412 log.cc:824] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:00.361615 28413 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Bootstrap starting.
I20260925 15:02:00.362030 28412 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: No bootstrap required, opened a new log
I20260925 15:02:00.362161 28413 tablet_bootstrap.cc:654] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:00.362102 28412 ts_tablet_manager.cc:1397] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260925 15:02:00.362409 28413 log.cc:824] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:00.363005 28413 tablet_bootstrap.cc:492] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: No bootstrap required, opened a new log
I20260925 15:02:00.363060 28413 ts_tablet_manager.cc:1397] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:00.363878 28411 raft_consensus.cc:348] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.364046 28411 raft_consensus.cc:374] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:00.364079 28411 raft_consensus.cc:729] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b40207091924470087669119d5c555bd, State: Initialized, Role: FOLLOWER
I20260925 15:02:00.364212 28411 consensus_queue.cc:260] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.364466 28411 ts_tablet_manager.cc:1428] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260925 15:02:00.364595 28266 heartbeater.cc:503] Master 127.8.13.254:41169 was elected leader, sending a full tablet report...
I20260925 15:02:00.364607 28412 raft_consensus.cc:348] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.364729 28412 raft_consensus.cc:374] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:00.364748 28412 raft_consensus.cc:729] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b322aeffc8d4018be42f3b3ee239b0d, State: Initialized, Role: FOLLOWER
I20260925 15:02:00.364818 28412 consensus_queue.cc:260] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.365001 28412 ts_tablet_manager.cc:1428] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260925 15:02:00.365052 28135 heartbeater.cc:503] Master 127.8.13.254:41169 was elected leader, sending a full tablet report...
I20260925 15:02:00.365108 28413 raft_consensus.cc:348] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.365260 28413 raft_consensus.cc:374] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:00.365293 28413 raft_consensus.cc:729] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31658c4cffb349f990ba7d81e36fca56, State: Initialized, Role: FOLLOWER
I20260925 15:02:00.365386 28413 consensus_queue.cc:260] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.365599 28413 ts_tablet_manager.cc:1428] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260925 15:02:00.365703 28397 heartbeater.cc:503] Master 127.8.13.254:41169 was elected leader, sending a full tablet report...
W20260925 15:02:00.461815 28267 tablet.cc:2369] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:00.532028 28417 raft_consensus.cc:482] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260925 15:02:00.532277 28417 raft_consensus.cc:504] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.532640 28417 leader_election.cc:288] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31658c4cffb349f990ba7d81e36fca56 (127.8.13.195:43563), 6b322aeffc8d4018be42f3b3ee239b0d (127.8.13.193:41891)
I20260925 15:02:00.536497 28408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.532915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:36697 (local address 127.8.13.193:41891)
0925 15:02:00.532958 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.532961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.533015 (+ 54us) server_negotiation.cc:407] Connection header received
0925 15:02:00.533102 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.533104 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.533115 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.533160 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:00.533302 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.533982 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.535492 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.536127 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.536193 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.536199 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.536377 (+ 178us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.536420 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:02:00.536497 28410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.532844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.13.194:55663 (local address 127.8.13.195:43563)
0925 15:02:00.532885 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.532887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.532896 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:02:00.532990 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.532991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.533001 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.533064 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0925 15:02:00.533218 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.533983 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.535491 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.536094 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.536180 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.536190 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.536377 (+ 187us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.536423 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:02:00.536525 28268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.532794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.195:43563 (local address 127.8.13.194:55663)
0925 15:02:00.532845 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.532855 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.532894 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.533076 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.533077 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.533078 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:00.533147 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.533149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.534086 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.534087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.535376 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.535380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.536200 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.536202 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.536291 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.536300 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.536365 (+ 65us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.536418 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":69}
I20260925 15:02:00.536530 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.532823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41891 (local address 127.8.13.194:36697)
0925 15:02:00.532965 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.532973 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:00.533025 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:00.533175 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:00.533176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:00.533177 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0925 15:02:00.533231 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.533233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.534097 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.534097 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.535376 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:00.535380 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.536215 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.536216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:00.536291 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.536300 (+ 9us) client_negotiation.cc:769] Sending connection context
0925 15:02:00.536364 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:02:00.536417 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":47,"thread_start_us":91,"threads_started":1}
I20260925 15:02:00.536903 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46243cc3702549d0a4690d9920e2e0a5" candidate_uuid: "b40207091924470087669119d5c555bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31658c4cffb349f990ba7d81e36fca56" is_pre_election: true
I20260925 15:02:00.536948 28084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46243cc3702549d0a4690d9920e2e0a5" candidate_uuid: "b40207091924470087669119d5c555bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" is_pre_election: true
I20260925 15:02:00.537091 28351 raft_consensus.cc:2454] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40207091924470087669119d5c555bd in term 0.
I20260925 15:02:00.537122 28084 raft_consensus.cc:2454] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b40207091924470087669119d5c555bd in term 0.
I20260925 15:02:00.537335 28154 leader_election.cc:302] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b322aeffc8d4018be42f3b3ee239b0d, b40207091924470087669119d5c555bd; no voters:
I20260925 15:02:00.537467 28417 raft_consensus.cc:2781] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260925 15:02:00.537518 28417 raft_consensus.cc:482] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260925 15:02:00.537552 28417 raft_consensus.cc:3037] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:00.538309 28417 raft_consensus.cc:504] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.538430 28417 leader_election.cc:288] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [CANDIDATE]: Term 1 election: Requested vote from peers 31658c4cffb349f990ba7d81e36fca56 (127.8.13.195:43563), 6b322aeffc8d4018be42f3b3ee239b0d (127.8.13.193:41891)
I20260925 15:02:00.538623 28084 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46243cc3702549d0a4690d9920e2e0a5" candidate_uuid: "b40207091924470087669119d5c555bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6b322aeffc8d4018be42f3b3ee239b0d"
I20260925 15:02:00.538707 28084 raft_consensus.cc:3037] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:00.538697 28351 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46243cc3702549d0a4690d9920e2e0a5" candidate_uuid: "b40207091924470087669119d5c555bd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31658c4cffb349f990ba7d81e36fca56"
I20260925 15:02:00.538784 28351 raft_consensus.cc:3037] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:00.539460 28084 raft_consensus.cc:2454] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40207091924470087669119d5c555bd in term 1.
I20260925 15:02:00.539602 28351 raft_consensus.cc:2454] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b40207091924470087669119d5c555bd in term 1.
I20260925 15:02:00.539633 28154 leader_election.cc:302] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6b322aeffc8d4018be42f3b3ee239b0d, b40207091924470087669119d5c555bd; no voters:
I20260925 15:02:00.539714 28417 raft_consensus.cc:2781] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:00.539850 28417 raft_consensus.cc:686] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 1 LEADER]: Becoming Leader. State: Replica: b40207091924470087669119d5c555bd, State: Running, Role: LEADER
I20260925 15:02:00.539949 28417 consensus_queue.cc:237] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } }
I20260925 15:02:00.540689 27947 catalog_manager.cc:5495] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd reported cstate change: term changed from 0 to 1, leader changed from <none> to b40207091924470087669119d5c555bd (127.8.13.194). New cstate: current_term: 1 leader_uuid: "b40207091924470087669119d5c555bd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b40207091924470087669119d5c555bd" member_type: VOTER last_known_addr { host: "127.8.13.194" port: 44677 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 } health_report { overall_health: UNKNOWN } } }
W20260925 15:02:00.574777 28398 tablet.cc:2369] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:00.581274 28407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.578904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.8.13.194:44677)
0925 15:02:00.578967 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:00.578969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:00.578978 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:02:00.579062 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:00.579063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:00.579080 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:00.579139 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:00.579264 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.579767 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.580555 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:00.580750 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:00.580804 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:00.580940 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:00.581048 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:00.581118 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:00.581156 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:02:00.581174 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260925 15:02:00.583014 28351 raft_consensus.cc:1261] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 1 FOLLOWER]: Refusing update from remote peer b40207091924470087669119d5c555bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:00.583012 28084 raft_consensus.cc:1261] T 46243cc3702549d0a4690d9920e2e0a5 P 6b322aeffc8d4018be42f3b3ee239b0d [term 1 FOLLOWER]: Refusing update from remote peer b40207091924470087669119d5c555bd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:00.583348 28421 consensus_queue.cc:1035] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "31658c4cffb349f990ba7d81e36fca56" member_type: VOTER last_known_addr { host: "127.8.13.195" port: 43563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:00.583477 28417 consensus_queue.cc:1035] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b322aeffc8d4018be42f3b3ee239b0d" member_type: VOTER last_known_addr { host: "127.8.13.193" port: 41891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:00.585942 28426 mvcc.cc:205] Tried to move back new op lower bound from 7333267540305494016 to 7333267540132212736. Current Snapshot: MvccSnapshot[applied={T|T < 7333267540305494016}]
I20260925 15:02:00.586103 28425 mvcc.cc:205] Tried to move back new op lower bound from 7333267540305494016 to 7333267540132212736. Current Snapshot: MvccSnapshot[applied={T|T < 7333267540305494016}]
I20260925 15:02:00.588136 27947 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:02:00.588258 27947 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260925 15:02:00.589027 27947 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8fcef4a99c74971b735e0130b015473: Sending DeleteTablet for 3 replicas of tablet 46243cc3702549d0a4690d9920e2e0a5
I20260925 15:02:00.589351 28331 tablet_service.cc:1514] Processing DeleteTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:00 UTC) from {username='slave'} at 127.0.0.1:53942
I20260925 15:02:00.589407 28200 tablet_service.cc:1514] Processing DeleteTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:00 UTC) from {username='slave'} at 127.0.0.1:38412
I20260925 15:02:00.589499 28068 tablet_service.cc:1514] Processing DeleteTablet for tablet 46243cc3702549d0a4690d9920e2e0a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-25 15:02:00 UTC) from {username='slave'} at 127.0.0.1:52778
I20260925 15:02:00.589771 28435 tablet_replica.cc:330] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: stopping tablet replica
I20260925 15:02:00.589814 28434 tablet_replica.cc:330] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: stopping tablet replica
I20260925 15:02:00.589862 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28007
I20260925 15:02:00.589912 28435 raft_consensus.cc:2229] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 1 LEADER]: Raft consensus shutting down.
I20260925 15:02:00.590029 28435 raft_consensus.cc:2258] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:02:00.590523 28435 ts_tablet_manager.cc:1905] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:02:00.591341 28434 raft_consensus.cc:2229] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260925 15:02:00.591492 28434 raft_consensus.cc:2258] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260925 15:02:00.591989 28434 ts_tablet_manager.cc:1905] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260925 15:02:00.592296 28435 ts_tablet_manager.cc:1918] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:02:00.592379 28435 log.cc:1196] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-1/wal/wals/46243cc3702549d0a4690d9920e2e0a5
I20260925 15:02:00.592669 28435 ts_tablet_manager.cc:1939] T 46243cc3702549d0a4690d9920e2e0a5 P b40207091924470087669119d5c555bd: Deleting consensus metadata
I20260925 15:02:00.593062 27934 catalog_manager.cc:4841] TS b40207091924470087669119d5c555bd (127.8.13.194:44677): tablet 46243cc3702549d0a4690d9920e2e0a5 (table test-table [id=ebb69fc91cf64df8b0adb18b0cb0d6c8]) successfully deleted
I20260925 15:02:00.593850 28434 ts_tablet_manager.cc:1918] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260925 15:02:00.593918 28434 log.cc:1196] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Deleting WAL directory at /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790348284423390-8247-0/minicluster-data/ts-2/wal/wals/46243cc3702549d0a4690d9920e2e0a5
I20260925 15:02:00.594122 28434 ts_tablet_manager.cc:1939] T 46243cc3702549d0a4690d9920e2e0a5 P 31658c4cffb349f990ba7d81e36fca56: Deleting consensus metadata
I20260925 15:02:00.594604 27933 catalog_manager.cc:4841] TS 31658c4cffb349f990ba7d81e36fca56 (127.8.13.195:43563): tablet 46243cc3702549d0a4690d9920e2e0a5 (table test-table [id=ebb69fc91cf64df8b0adb18b0cb0d6c8]) successfully deleted
W20260925 15:02:00.596233 27932 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.13.193:41891 (error 108)
I20260925 15:02:00.596354 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28138
I20260925 15:02:00.598088 28409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:00.597914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.193:41891 (local address 127.0.0.1:52784)
0925 15:02:00.597981 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:00.598046 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260925 15:02:00.598261 27932 catalog_manager.cc:4584] TS 6b322aeffc8d4018be42f3b3ee239b0d (127.8.13.193:41891): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 46243cc3702549d0a4690d9920e2e0a5: Network error: Client connection negotiation failed: client connection to 127.8.13.193:41891: connect: Connection refused (error 111)
I20260925 15:02:00.600466 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28269
I20260925 15:02:00.604885 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 27916
2026-09-25T15:02:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (822 ms)
[----------] 6 tests from EncryptionPolicyTest (7763 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:00 dist-test-slave-15hz krb5kdc[28440](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:00 dist-test-slave-15hz krb5kdc[28440](info): set up 2 sockets
Sep 25 15:02:00 dist-test-slave-15hz krb5kdc[28440](info): commencing operation
krb5kdc: starting...
W20260925 15:02:01.551198 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:01.669872 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:01.751075 28453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:01.751303 28453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:01.751335 28453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:01.752578 28453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:01.752624 28453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:01.752641 28453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:01.752656 28453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:01.752671 28453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:01.754235 28453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28453
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:01.754495 28453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:01.754729 28453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:01.759389 28461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:01.759385 28458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:01.759415 28459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:01.759786 28453 server_base.cc:1034] running on GCE node
I20260925 15:02:01.760012 28453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:01.760309 28453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:01.761459 28453 hybrid_clock.cc:648] HybridClock initialized: now 1790348521761444 us; error 30 us; skew 500 ppm
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28440](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348521, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:01.764154 28453 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:01.764616 28453 webserver.cc:460] Webserver started at http://127.0.0.1:44369/ using document root <none> and password file <none>
I20260925 15:02:01.764861 28453 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:01.764909 28453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:01.765038 28453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:01.766139 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "5f0ae67e2d464095b2c32edad3d03631"
format_stamp: "Formatted at 2026-09-25 15:02:01 on dist-test-slave-15hz"
server_key: "506c66ee9cfd9bd8ddf733b6de835699"
server_key_iv: "07e904e0f8df94feb721e3774ad9ca5f"
server_key_version: "encryptionkey@0"
I20260925 15:02:01.766489 28453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "5f0ae67e2d464095b2c32edad3d03631"
format_stamp: "Formatted at 2026-09-25 15:02:01 on dist-test-slave-15hz"
server_key: "506c66ee9cfd9bd8ddf733b6de835699"
server_key_iv: "07e904e0f8df94feb721e3774ad9ca5f"
server_key_version: "encryptionkey@0"
I20260925 15:02:01.767853 28453 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:01.768642 28468 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:01.768827 28453 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:01.768918 28453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "5f0ae67e2d464095b2c32edad3d03631"
format_stamp: "Formatted at 2026-09-25 15:02:01 on dist-test-slave-15hz"
server_key: "506c66ee9cfd9bd8ddf733b6de835699"
server_key_iv: "07e904e0f8df94feb721e3774ad9ca5f"
server_key_version: "encryptionkey@0"
I20260925 15:02:01.768985 28453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:01.780086 28453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:01.782104 28453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:01.782243 28453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:01.786074 28453 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45369
I20260925 15:02:01.786123 28520 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45369 every 8 connection(s)
I20260925 15:02:01.786442 28453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:01.787392 28521 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:01.789716 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: Bootstrap starting.
I20260925 15:02:01.790292 28521 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:01.790555 28521 log.cc:824] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:01.791201 28521 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: No bootstrap required, opened a new log
I20260925 15:02:01.792825 28521 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 15:02:01.792971 28521 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:01.792991 28521 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0ae67e2d464095b2c32edad3d03631, State: Initialized, Role: FOLLOWER
I20260925 15:02:01.793136 28521 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 15:02:01.793208 28521 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:01.793232 28521 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:01.793279 28521 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:01.793800 28521 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 15:02:01.793892 28521 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0ae67e2d464095b2c32edad3d03631; no voters:
I20260925 15:02:01.794060 28521 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:01.794093 28524 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:01.794248 28524 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [term 1 LEADER]: Becoming Leader. State: Replica: 5f0ae67e2d464095b2c32edad3d03631, State: Running, Role: LEADER
I20260925 15:02:01.794314 28521 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:01.794492 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28453
I20260925 15:02:01.794615 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:01.794800 8247 external_mini_cluster.cc:1428] Setting key 7a464cc4b6d7b1f2f7dd199cf4a97cb3
I20260925 15:02:01.794974 28524 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } }
I20260925 15:02:01.795589 28526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0ae67e2d464095b2c32edad3d03631. Latest consensus state: current_term: 1 leader_uuid: "5f0ae67e2d464095b2c32edad3d03631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260925 15:02:01.795683 28526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:01.795776 28525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0ae67e2d464095b2c32edad3d03631" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0ae67e2d464095b2c32edad3d03631" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45369 } } }
I20260925 15:02:01.796034 28525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631 [sys.catalog]: This master's current role is: LEADER
W20260925 15:02:01.796155 28540 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:01.796201 28540 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:01.796270 28542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:01.796460 28542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:01.797981 28542 catalog_manager.cc:1347] Generated new cluster ID: d0a8e27605b04933b1715fefc37bfd2c
I20260925 15:02:01.798034 28542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28440](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348521, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:02:01.806099 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:01.795442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39395 (local address 127.0.0.1:45369)
0925 15:02:01.795617 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:01.795626 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:01.796065 (+ 439us) server_negotiation.cc:407] Connection header received
0925 15:02:01.796234 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:01.796242 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:01.796305 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:01.796437 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:01.797033 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:01.798293 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:01.799016 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:01.799456 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:01.799567 (+ 111us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:01.801875 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:01.801880 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:01.801880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:01.801892 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:01.804444 (+ 2552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:01.804890 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:01.804893 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:01.804894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:01.805008 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:01.805232 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:01.805233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:01.805234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:01.805499 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:01.805675 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:01.805766 (+ 91us) server_negotiation.cc:299] Negotiation successful
0925 15:02:01.805876 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":62,"threads_started":1}
I20260925 15:02:01.807104 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:01.814769 28542 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:01.815493 28542 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:01.818324 28538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:01.811885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58153 (local address 127.0.0.1:45369)
0925 15:02:01.811929 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:01.811931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:01.812121 (+ 190us) server_negotiation.cc:407] Connection header received
0925 15:02:01.812181 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:01.812181 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:01.812193 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:01.812250 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:01.812819 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:01.813544 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:01.814284 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:01.814615 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:01.814699 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:01.815988 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:01.815989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:01.815990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:01.815997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:01.817247 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:01.817595 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:01.817597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:01.817597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:01.817651 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:01.817843 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:01.817843 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:01.817844 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:01.817951 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:01.818053 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:01.818161 (+ 108us) server_negotiation.cc:299] Negotiation successful
0925 15:02:01.818229 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":19}
I20260925 15:02:01.820606 28542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0ae67e2d464095b2c32edad3d03631: Generated new TSK 0
I20260925 15:02:01.820780 28542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:01.824915 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28453
W20260925 15:02:01.829048 28438 connection.cc:535] client connection to 127.0.0.1:45369 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45369): Connection reset by peer (error 104)
2026-09-25T15:02:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1218 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28552](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28552](info): set up 2 sockets
Sep 25 15:02:01 dist-test-slave-15hz krb5kdc[28552](info): commencing operation
krb5kdc: starting...
W20260925 15:02:02.767400 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:02 dist-test-slave-15hz krb5kdc[28552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348522, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:02.882160 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39755
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39755
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:02.962880 28565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:02.963104 28565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:02.963124 28565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:02.964289 28565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:02.964331 28565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:02.964344 28565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:02.964355 28565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:02.964366 28565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:02.965807 28565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39755
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39755
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:02.966017 28565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:02.966246 28565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:02.971043 28570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:02.971052 28571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:02.971064 28573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:02.971223 28565 server_base.cc:1034] running on GCE node
I20260925 15:02:02.971573 28565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:02.971886 28565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:02.973089 28565 hybrid_clock.cc:648] HybridClock initialized: now 1790348522973070 us; error 31 us; skew 500 ppm
Sep 25 15:02:02 dist-test-slave-15hz krb5kdc[28552](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348522, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:02.975560 28565 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:02.976047 28565 webserver.cc:460] Webserver started at http://127.0.0.1:33479/ using document root <none> and password file <none>
I20260925 15:02:02.976267 28565 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:02.976315 28565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:02.976442 28565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:02.977516 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "e1d54edcc3c24c55a3e47ba4437765e6"
format_stamp: "Formatted at 2026-09-25 15:02:02 on dist-test-slave-15hz"
server_key: "5122be631365f365295dd230fb25d317"
server_key_iv: "9e970fc8568a01908f09ad241f987eb1"
server_key_version: "encryptionkey@0"
I20260925 15:02:02.977851 28565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "e1d54edcc3c24c55a3e47ba4437765e6"
format_stamp: "Formatted at 2026-09-25 15:02:02 on dist-test-slave-15hz"
server_key: "5122be631365f365295dd230fb25d317"
server_key_iv: "9e970fc8568a01908f09ad241f987eb1"
server_key_version: "encryptionkey@0"
I20260925 15:02:02.979153 28565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:02.979982 28580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:02.980160 28565 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:02:02.980247 28565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "e1d54edcc3c24c55a3e47ba4437765e6"
format_stamp: "Formatted at 2026-09-25 15:02:02 on dist-test-slave-15hz"
server_key: "5122be631365f365295dd230fb25d317"
server_key_iv: "9e970fc8568a01908f09ad241f987eb1"
server_key_version: "encryptionkey@0"
I20260925 15:02:02.980336 28565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:02.990770 28565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:02.992875 28565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:02.993012 28565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:02.996827 28565 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39755
I20260925 15:02:02.996867 28632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39755 every 8 connection(s)
I20260925 15:02:02.997207 28565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:02.998142 28633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:03.000464 28633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6: Bootstrap starting.
I20260925 15:02:03.001008 28633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:03.001255 28633 log.cc:824] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:03.001893 28633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6: No bootstrap required, opened a new log
I20260925 15:02:03.003404 28633 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } }
I20260925 15:02:03.003578 28633 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:03.003626 28633 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1d54edcc3c24c55a3e47ba4437765e6, State: Initialized, Role: FOLLOWER
I20260925 15:02:03.003775 28633 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } }
I20260925 15:02:03.003839 28633 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:03.003870 28633 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:03.003916 28633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:03.004418 28633 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } }
I20260925 15:02:03.004519 28633 leader_election.cc:302] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1d54edcc3c24c55a3e47ba4437765e6; no voters:
I20260925 15:02:03.004683 28633 leader_election.cc:288] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:03.004715 28636 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:03.004832 28636 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [term 1 LEADER]: Becoming Leader. State: Replica: e1d54edcc3c24c55a3e47ba4437765e6, State: Running, Role: LEADER
I20260925 15:02:03.004912 28636 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } }
I20260925 15:02:03.004945 28633 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:03.005304 28637 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } } }
I20260925 15:02:03.005383 28637 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:03.005434 28638 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1d54edcc3c24c55a3e47ba4437765e6. Latest consensus state: current_term: 1 leader_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1d54edcc3c24c55a3e47ba4437765e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39755 } } }
I20260925 15:02:03.005549 28638 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:03.005963 28642 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:03.006513 28642 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:03.007095 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28565
I20260925 15:02:03.007217 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:03.007403 8247 external_mini_cluster.cc:1428] Setting key 7b089449394fd94f0377f81ad10ff93d
I20260925 15:02:03.008095 28642 catalog_manager.cc:1347] Generated new cluster ID: 6998bc2ecb6e4f6c8cb0825b53d0185d
I20260925 15:02:03.008147 28642 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:03.021181 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:03.007875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60453 (local address 127.0.0.1:39755)
0925 15:02:03.008017 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:03.008027 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:03.011792 (+ 3765us) server_negotiation.cc:407] Connection header received
0925 15:02:03.011929 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:03.011938 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:03.011997 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:03.012174 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:03.012382 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:03.014198 (+ 1816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:03.015328 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:03.016005 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:03.016132 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:03.020344 (+ 4212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:03.020350 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:03.020350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:03.020605 (+ 255us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:03.020653 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:03.020680 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:03.020784 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:03.020871 (+ 87us) server_negotiation.cc:299] Negotiation successful
0925 15:02:03.020908 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
I20260925 15:02:03.022245 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:03.027503 28656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:03.023772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38439 (local address 127.0.0.1:39755)
0925 15:02:03.023867 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:03.023869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:03.023922 (+ 53us) server_negotiation.cc:407] Connection header received
0925 15:02:03.024002 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:03.024003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:03.024014 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:03.024076 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:03.024213 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:03.024973 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:03.025985 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:03.026329 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:03.026394 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:03.027220 (+ 826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:03.027222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:03.027222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:03.027227 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:03.027242 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:03.027247 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:03.027280 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:03.027400 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:02:03.027415 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260925 15:02:03.028983 28642 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:03.029523 28642 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:03.035631 28642 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1d54edcc3c24c55a3e47ba4437765e6: Generated new TSK 0
I20260925 15:02:03.035866 28642 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:03.040962 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28565
W20260925 15:02:03.044684 28550 connection.cc:535] client connection to 127.0.0.1:39755 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39755): Connection reset by peer (error 104)
2026-09-25T15:02:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:03 dist-test-slave-15hz krb5kdc[28664](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:03 dist-test-slave-15hz krb5kdc[28664](info): set up 2 sockets
Sep 25 15:02:03 dist-test-slave-15hz krb5kdc[28664](info): commencing operation
krb5kdc: starting...
W20260925 15:02:03.987263 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:04 dist-test-slave-15hz krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348524, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:04.099853 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36395
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:04.180015 28677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:04.180234 28677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:04.180258 28677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:04.181443 28677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:04.181488 28677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:04.181501 28677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:04.181512 28677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:04.181523 28677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:04.183019 28677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36395
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:04.183255 28677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:04.183468 28677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:04.188261 28685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:04.188247 28682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:04.188234 28683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:04.188385 28677 server_base.cc:1034] running on GCE node
I20260925 15:02:04.188637 28677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:04.188930 28677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:04.190106 28677 hybrid_clock.cc:648] HybridClock initialized: now 1790348524190096 us; error 36 us; skew 500 ppm
Sep 25 15:02:04 dist-test-slave-15hz krb5kdc[28664](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348524, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:04.192684 28677 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:04.193194 28677 webserver.cc:460] Webserver started at http://127.0.0.1:41549/ using document root <none> and password file <none>
I20260925 15:02:04.193413 28677 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:04.193467 28677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:04.193622 28677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:04.194809 28677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "b1e894bf1b034036a9a0d65ba3d11a27"
format_stamp: "Formatted at 2026-09-25 15:02:04 on dist-test-slave-15hz"
server_key: "91a8204bbea44131bc1ba9aee197fd46"
server_key_iv: "ed484b114360ffda636adcf6bc9ee565"
server_key_version: "encryptionkey@0"
I20260925 15:02:04.195173 28677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "b1e894bf1b034036a9a0d65ba3d11a27"
format_stamp: "Formatted at 2026-09-25 15:02:04 on dist-test-slave-15hz"
server_key: "91a8204bbea44131bc1ba9aee197fd46"
server_key_iv: "ed484b114360ffda636adcf6bc9ee565"
server_key_version: "encryptionkey@0"
I20260925 15:02:04.196636 28677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:02:04.197340 28692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:04.197486 28677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:04.197582 28677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "b1e894bf1b034036a9a0d65ba3d11a27"
format_stamp: "Formatted at 2026-09-25 15:02:04 on dist-test-slave-15hz"
server_key: "91a8204bbea44131bc1ba9aee197fd46"
server_key_iv: "ed484b114360ffda636adcf6bc9ee565"
server_key_version: "encryptionkey@0"
I20260925 15:02:04.197647 28677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:04.201239 28677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:04.203080 28677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:04.203219 28677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:04.207135 28677 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37551
I20260925 15:02:04.207170 28744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37551 every 8 connection(s)
I20260925 15:02:04.207490 28677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:04.208439 28745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:04.210669 28745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: Bootstrap starting.
I20260925 15:02:04.211241 28745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:04.211484 28745 log.cc:824] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:04.212173 28745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: No bootstrap required, opened a new log
I20260925 15:02:04.213758 28745 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 15:02:04.213904 28745 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:04.213925 28745 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1e894bf1b034036a9a0d65ba3d11a27, State: Initialized, Role: FOLLOWER
I20260925 15:02:04.214061 28745 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 15:02:04.214133 28745 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:04.214155 28745 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:04.214185 28745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:04.214459 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28677
I20260925 15:02:04.214569 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:04.214735 8247 external_mini_cluster.cc:1428] Setting key bb820a61948e6b1b96318384cbbdd76c
I20260925 15:02:04.214751 28745 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 15:02:04.214851 28745 leader_election.cc:302] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1e894bf1b034036a9a0d65ba3d11a27; no voters:
I20260925 15:02:04.215025 28745 leader_election.cc:288] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:04.215134 28748 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:04.215312 28748 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [term 1 LEADER]: Becoming Leader. State: Replica: b1e894bf1b034036a9a0d65ba3d11a27, State: Running, Role: LEADER
I20260925 15:02:04.215317 28745 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:04.216380 28748 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } }
I20260925 15:02:04.216962 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:04.215250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39953 (local address 127.0.0.1:37551)
0925 15:02:04.215448 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:04.215457 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:04.215488 (+ 31us) server_negotiation.cc:407] Connection header received
0925 15:02:04.215623 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:04.215630 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:04.215656 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:04.215773 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:04.216438 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:04.216442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:04.216442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:04.216446 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:04.216491 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:04.216505 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:04.216593 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:04.216724 (+ 131us) server_negotiation.cc:299] Negotiation successful
0925 15:02:04.216770 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260925 15:02:04.217093 28751 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260925 15:02:04.217206 28751 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:04.217092 28749 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1e894bf1b034036a9a0d65ba3d11a27. Latest consensus state: current_term: 1 leader_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1e894bf1b034036a9a0d65ba3d11a27" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37551 } } }
I20260925 15:02:04.217344 28749 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27 [sys.catalog]: This master's current role is: LEADER
W20260925 15:02:04.217622 28764 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:04.217671 28764 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:04.217720 28765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:04.217860 28765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:04.217929 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:04.219141 28765 catalog_manager.cc:1347] Generated new cluster ID: 23999c7536474946b41c14442273e1bd
I20260925 15:02:04.219187 28765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:04.227367 28765 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:04.227602 28765 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:04.228363 28752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:04.219851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38739 (local address 127.0.0.1:37551)
0925 15:02:04.219886 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:04.219888 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:04.227261 (+ 7373us) server_negotiation.cc:407] Connection header received
0925 15:02:04.227332 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:04.227333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:04.227340 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:04.227415 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:04.228149 (+ 734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:04.228151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:04.228151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:04.228156 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:04.228172 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:04.228178 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:04.228228 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:04.228275 (+ 47us) server_negotiation.cc:299] Negotiation successful
0925 15:02:04.228290 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:02:04.242401 28765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1e894bf1b034036a9a0d65ba3d11a27: Generated new TSK 0
I20260925 15:02:04.242720 28765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:04.248797 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28677
2026-09-25T15:02:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:04 dist-test-slave-15hz krb5kdc[28776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:04 dist-test-slave-15hz krb5kdc[28776](info): set up 2 sockets
Sep 25 15:02:04 dist-test-slave-15hz krb5kdc[28776](info): commencing operation
krb5kdc: starting...
W20260925 15:02:05.199306 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:05.314553 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:05.395709 28789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:05.395960 28789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:05.395998 28789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:05.397320 28789 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:05.397382 28789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:05.397399 28789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:05.397415 28789 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:05.397430 28789 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:05.399114 28789 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:05.399403 28789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:05.399703 28789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:05.404594 28797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:05.404616 28795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:05.404616 28794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:05.404690 28789 server_base.cc:1034] running on GCE node
I20260925 15:02:05.405087 28789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:05.405385 28789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:05.406584 28789 hybrid_clock.cc:648] HybridClock initialized: now 1790348525406561 us; error 38 us; skew 500 ppm
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348525, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:05.409274 28789 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:05.409772 28789 webserver.cc:460] Webserver started at http://127.0.0.1:42837/ using document root <none> and password file <none>
I20260925 15:02:05.409978 28789 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:05.410037 28789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:05.410166 28789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:05.411237 28789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "af2ec341637e44c285c34063d56b70c0"
format_stamp: "Formatted at 2026-09-25 15:02:05 on dist-test-slave-15hz"
server_key: "00a56acddaf25fbecb8b4d86d4456e1a"
server_key_iv: "c828e9ae5cdd0fe15e3b40461f0df740"
server_key_version: "encryptionkey@0"
I20260925 15:02:05.411619 28789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "af2ec341637e44c285c34063d56b70c0"
format_stamp: "Formatted at 2026-09-25 15:02:05 on dist-test-slave-15hz"
server_key: "00a56acddaf25fbecb8b4d86d4456e1a"
server_key_iv: "c828e9ae5cdd0fe15e3b40461f0df740"
server_key_version: "encryptionkey@0"
I20260925 15:02:05.413002 28789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:05.413766 28804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:05.413916 28789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:05.414002 28789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "af2ec341637e44c285c34063d56b70c0"
format_stamp: "Formatted at 2026-09-25 15:02:05 on dist-test-slave-15hz"
server_key: "00a56acddaf25fbecb8b4d86d4456e1a"
server_key_iv: "c828e9ae5cdd0fe15e3b40461f0df740"
server_key_version: "encryptionkey@0"
I20260925 15:02:05.414108 28789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:05.428817 28789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:05.430742 28789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:05.430882 28789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:05.435132 28789 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37675
I20260925 15:02:05.435238 28856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37675 every 8 connection(s)
I20260925 15:02:05.435573 28789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:05.436543 28857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:05.438838 28857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0: Bootstrap starting.
I20260925 15:02:05.439409 28857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:05.439414 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28789
I20260925 15:02:05.439517 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:05.439704 8247 external_mini_cluster.cc:1428] Setting key 2a8f40e7f0d87594e1a167acfe6f4430
I20260925 15:02:05.439705 28857 log.cc:824] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:05.440474 28857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0: No bootstrap required, opened a new log
I20260925 15:02:05.442994 28857 raft_consensus.cc:348] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260925 15:02:05.443179 28857 raft_consensus.cc:374] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:05.443218 28857 raft_consensus.cc:729] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af2ec341637e44c285c34063d56b70c0, State: Initialized, Role: FOLLOWER
I20260925 15:02:05.443357 28857 consensus_queue.cc:260] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260925 15:02:05.443430 28857 raft_consensus.cc:388] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:05.443462 28857 raft_consensus.cc:482] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:05.443511 28857 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:05.444195 28857 raft_consensus.cc:504] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260925 15:02:05.444305 28857 leader_election.cc:302] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af2ec341637e44c285c34063d56b70c0; no voters:
I20260925 15:02:05.444480 28857 leader_election.cc:288] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:05.444523 28862 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:05.444665 28862 raft_consensus.cc:686] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [term 1 LEADER]: Becoming Leader. State: Replica: af2ec341637e44c285c34063d56b70c0, State: Running, Role: LEADER
I20260925 15:02:05.444773 28857 sys_catalog.cc:564] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:05.444809 28862 consensus_queue.cc:237] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } }
I20260925 15:02:05.445158 28864 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af2ec341637e44c285c34063d56b70c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260925 15:02:05.445238 28864 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:05.445472 28862 sys_catalog.cc:455] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af2ec341637e44c285c34063d56b70c0. Latest consensus state: current_term: 1 leader_uuid: "af2ec341637e44c285c34063d56b70c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af2ec341637e44c285c34063d56b70c0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37675 } } }
I20260925 15:02:05.445590 28862 sys_catalog.cc:458] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:05.446050 28869 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:05.446667 28869 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348525, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:02:05.448228 28869 catalog_manager.cc:1347] Generated new cluster ID: 9fa4734f2d4d45d7989b8bd0806f1087
I20260925 15:02:05.448278 28869 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:05.451401 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:05.440227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53851 (local address 127.0.0.1:37675)
0925 15:02:05.440383 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:05.440393 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:05.440426 (+ 33us) server_negotiation.cc:407] Connection header received
0925 15:02:05.440944 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:05.440954 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:05.441024 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:05.441186 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:05.441731 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:05.443135 (+ 1404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:05.444045 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:05.444673 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:05.444780 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:05.447493 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:05.447497 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:05.447498 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:05.447510 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:05.449398 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:05.449817 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:05.449820 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:05.449821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:05.449923 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:05.450120 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:05.450121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:05.450121 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:05.450424 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:05.450571 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:05.451040 (+ 469us) server_negotiation.cc:299] Negotiation successful
0925 15:02:05.451180 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
I20260925 15:02:05.452330 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:05.460791 28861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:05.453848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36131 (local address 127.0.0.1:37675)
0925 15:02:05.453883 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:05.453885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:05.453952 (+ 67us) server_negotiation.cc:407] Connection header received
0925 15:02:05.454018 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:05.454019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:05.454029 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:05.454080 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:05.454661 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:05.455418 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:05.456439 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:05.456774 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:05.456847 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:05.458166 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:05.458168 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:05.458168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:05.458175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:05.459719 (+ 1544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:05.460076 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:05.460077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:05.460077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:05.460136 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:05.460301 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:05.460301 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:05.460302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:05.460429 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:05.460513 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:05.460606 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:02:05.460690 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:05.479822 28869 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:05.480587 28869 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:05.489734 28869 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af2ec341637e44c285c34063d56b70c0: Generated new TSK 0
I20260925 15:02:05.489940 28869 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:05.497148 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28789
2026-09-25T15:02:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28888](info): set up 2 sockets
Sep 25 15:02:05 dist-test-slave-15hz krb5kdc[28888](info): commencing operation
krb5kdc: starting...
W20260925 15:02:06.439507 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:06 dist-test-slave-15hz krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348526, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:06.550575 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:06.632141 28901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:06.632423 28901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:06.632457 28901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:06.634145 28901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:06.634202 28901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:06.634222 28901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:06.634238 28901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:06.634253 28901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:06.636031 28901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:06.636337 28901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:06.636571 28901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:06.641360 28909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:06.641366 28906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:06.641554 28901 server_base.cc:1034] running on GCE node
W20260925 15:02:06.641355 28907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:06.641865 28901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:06.642140 28901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:06.643280 28901 hybrid_clock.cc:648] HybridClock initialized: now 1790348526643266 us; error 28 us; skew 500 ppm
Sep 25 15:02:06 dist-test-slave-15hz krb5kdc[28888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348526, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:06.645670 28901 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:06.646073 28901 webserver.cc:460] Webserver started at http://127.0.0.1:36251/ using document root <none> and password file <none>
I20260925 15:02:06.646284 28901 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:06.646334 28901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:06.646458 28901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:06.647483 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "ebef55af26fe46ed9e0c3602be13ce59"
format_stamp: "Formatted at 2026-09-25 15:02:06 on dist-test-slave-15hz"
server_key: "6b863f8e6f2559a418f449d2bbc3a3e7"
server_key_iv: "370d5ce9842e54c3c4e417d57ebb74a5"
server_key_version: "encryptionkey@0"
I20260925 15:02:06.647858 28901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "ebef55af26fe46ed9e0c3602be13ce59"
format_stamp: "Formatted at 2026-09-25 15:02:06 on dist-test-slave-15hz"
server_key: "6b863f8e6f2559a418f449d2bbc3a3e7"
server_key_iv: "370d5ce9842e54c3c4e417d57ebb74a5"
server_key_version: "encryptionkey@0"
I20260925 15:02:06.649278 28901 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:06.649940 28916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:06.650076 28901 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:06.650158 28901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "ebef55af26fe46ed9e0c3602be13ce59"
format_stamp: "Formatted at 2026-09-25 15:02:06 on dist-test-slave-15hz"
server_key: "6b863f8e6f2559a418f449d2bbc3a3e7"
server_key_iv: "370d5ce9842e54c3c4e417d57ebb74a5"
server_key_version: "encryptionkey@0"
I20260925 15:02:06.650226 28901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:06.693356 28901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:06.695359 28901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:06.695510 28901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:06.699251 28901 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39661
I20260925 15:02:06.699304 28968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39661 every 8 connection(s)
I20260925 15:02:06.699661 28901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:06.700619 28969 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:06.703125 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59: Bootstrap starting.
I20260925 15:02:06.703795 28969 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:06.704067 28969 log.cc:824] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:06.704834 28969 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59: No bootstrap required, opened a new log
I20260925 15:02:06.705559 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 28901
I20260925 15:02:06.705665 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:06.705825 8247 external_mini_cluster.cc:1428] Setting key 41ac15a4450f738e32de63f891e989cd
I20260925 15:02:06.706714 28969 raft_consensus.cc:348] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 15:02:06.706880 28969 raft_consensus.cc:374] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:06.706918 28969 raft_consensus.cc:729] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebef55af26fe46ed9e0c3602be13ce59, State: Initialized, Role: FOLLOWER
I20260925 15:02:06.707050 28969 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 15:02:06.707132 28969 raft_consensus.cc:388] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:06.707176 28969 raft_consensus.cc:482] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:06.707231 28969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:06.707844 28969 raft_consensus.cc:504] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 15:02:06.707944 28969 leader_election.cc:302] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebef55af26fe46ed9e0c3602be13ce59; no voters:
I20260925 15:02:06.708110 28969 leader_election.cc:288] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:06.708146 28974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:06.708285 28974 raft_consensus.cc:686] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [term 1 LEADER]: Becoming Leader. State: Replica: ebef55af26fe46ed9e0c3602be13ce59, State: Running, Role: LEADER
I20260925 15:02:06.708437 28969 sys_catalog.cc:564] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:06.708424 28974 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } }
I20260925 15:02:06.708783 28974 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebef55af26fe46ed9e0c3602be13ce59. Latest consensus state: current_term: 1 leader_uuid: "ebef55af26fe46ed9e0c3602be13ce59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260925 15:02:06.708871 28974 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:06.709445 28980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:06.710001 28980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:06.710359 28975 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebef55af26fe46ed9e0c3602be13ce59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebef55af26fe46ed9e0c3602be13ce59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39661 } } }
I20260925 15:02:06.710505 28975 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:06.711637 28980 catalog_manager.cc:1347] Generated new cluster ID: f7020e824d0b426aab7fb4b3784d1aa3
I20260925 15:02:06.711715 28980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:06.711802 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:06.706327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35821 (local address 127.0.0.1:39661)
0925 15:02:06.706485 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:06.706492 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:06.706517 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:02:06.706666 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:06.706671 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:06.706721 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:06.706829 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:06.707088 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:06.708384 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:06.709403 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:06.709968 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:06.710063 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:06.711047 (+ 984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:06.711050 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:06.711051 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:06.711267 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:06.711305 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:06.711317 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:06.711373 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:06.711445 (+ 72us) server_negotiation.cc:299] Negotiation successful
0925 15:02:06.711507 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260925 15:02:06.712589 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:06.716361 28973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:06.713762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58427 (local address 127.0.0.1:39661)
0925 15:02:06.713797 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:06.713799 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:06.713916 (+ 117us) server_negotiation.cc:407] Connection header received
0925 15:02:06.713969 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:06.713970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:06.713980 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:06.714022 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:06.714144 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:06.714602 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:06.715309 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:06.715502 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:06.715580 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:06.716149 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:06.716151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:06.716151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:06.716155 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:06.716169 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:06.716172 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:06.716223 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:06.716274 (+ 51us) server_negotiation.cc:299] Negotiation successful
0925 15:02:06.716295 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:06.732816 28980 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:06.733614 28980 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:06.747808 28980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ebef55af26fe46ed9e0c3602be13ce59: Generated new TSK 0
I20260925 15:02:06.748054 28980 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:06.754323 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 28901
2026-09-25T15:02:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1256 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:06 dist-test-slave-15hz krb5kdc[29000](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:06 dist-test-slave-15hz krb5kdc[29000](info): set up 2 sockets
Sep 25 15:02:06 dist-test-slave-15hz krb5kdc[29000](info): commencing operation
krb5kdc: starting...
W20260925 15:02:07.691396 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:07 dist-test-slave-15hz krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:07.803103 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38193
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38193
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:07.884557 29013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:07.884778 29013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:07.884810 29013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:07.885990 29013 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:07.886046 29013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:07.886065 29013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:07.886080 29013 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:07.886096 29013 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:07.887645 29013 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38193
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:07.887898 29013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:07.888113 29013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:07.892805 29021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:07.892844 29013 server_base.cc:1034] running on GCE node
W20260925 15:02:07.892874 29019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:07.892908 29018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:07.893260 29013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:07.893525 29013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:07.894690 29013 hybrid_clock.cc:648] HybridClock initialized: now 1790348527894665 us; error 36 us; skew 500 ppm
Sep 25 15:02:07 dist-test-slave-15hz krb5kdc[29000](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348527, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:07.897197 29013 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:07.897676 29013 webserver.cc:460] Webserver started at http://127.0.0.1:45999/ using document root <none> and password file <none>
I20260925 15:02:07.897882 29013 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:07.897929 29013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:07.898052 29013 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:07.899112 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "c67956db999547a5aef2a1a23f7b54e2"
format_stamp: "Formatted at 2026-09-25 15:02:07 on dist-test-slave-15hz"
server_key: "e0c6e6a552d1e7fa2f0aa1d7be54a9bc"
server_key_iv: "a2a53ade6e112ccf841fb01d4d09d24e"
server_key_version: "encryptionkey@0"
I20260925 15:02:07.899482 29013 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "c67956db999547a5aef2a1a23f7b54e2"
format_stamp: "Formatted at 2026-09-25 15:02:07 on dist-test-slave-15hz"
server_key: "e0c6e6a552d1e7fa2f0aa1d7be54a9bc"
server_key_iv: "a2a53ade6e112ccf841fb01d4d09d24e"
server_key_version: "encryptionkey@0"
I20260925 15:02:07.900799 29013 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:07.901511 29028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:07.901665 29013 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:07.901758 29013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "c67956db999547a5aef2a1a23f7b54e2"
format_stamp: "Formatted at 2026-09-25 15:02:07 on dist-test-slave-15hz"
server_key: "e0c6e6a552d1e7fa2f0aa1d7be54a9bc"
server_key_iv: "a2a53ade6e112ccf841fb01d4d09d24e"
server_key_version: "encryptionkey@0"
I20260925 15:02:07.901829 29013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:07.905148 29013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:07.907226 29013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:07.907337 29013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:07.911073 29013 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38193
I20260925 15:02:07.911114 29080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38193 every 8 connection(s)
I20260925 15:02:07.911419 29013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:07.912340 29081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:07.914649 29081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2: Bootstrap starting.
I20260925 15:02:07.915199 29081 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:07.915447 29081 log.cc:824] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:07.916098 29081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2: No bootstrap required, opened a new log
I20260925 15:02:07.917696 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29013
I20260925 15:02:07.917652 29081 raft_consensus.cc:348] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } }
I20260925 15:02:07.917804 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:07.917794 29081 raft_consensus.cc:374] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:07.917831 29081 raft_consensus.cc:729] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c67956db999547a5aef2a1a23f7b54e2, State: Initialized, Role: FOLLOWER
I20260925 15:02:07.917979 8247 external_mini_cluster.cc:1428] Setting key caeccc8f78fbcdd005208bfd947e8396
I20260925 15:02:07.917973 29081 consensus_queue.cc:260] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } }
I20260925 15:02:07.918035 29081 raft_consensus.cc:388] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:07.918071 29081 raft_consensus.cc:482] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:07.918118 29081 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:07.918751 29081 raft_consensus.cc:504] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } }
I20260925 15:02:07.918843 29081 leader_election.cc:302] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c67956db999547a5aef2a1a23f7b54e2; no voters:
I20260925 15:02:07.918998 29081 leader_election.cc:288] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:07.919014 29086 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:07.919121 29086 raft_consensus.cc:686] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [term 1 LEADER]: Becoming Leader. State: Replica: c67956db999547a5aef2a1a23f7b54e2, State: Running, Role: LEADER
I20260925 15:02:07.919211 29086 consensus_queue.cc:237] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } }
I20260925 15:02:07.919248 29081 sys_catalog.cc:564] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:07.919569 29087 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c67956db999547a5aef2a1a23f7b54e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } } }
I20260925 15:02:07.919608 29088 sys_catalog.cc:455] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c67956db999547a5aef2a1a23f7b54e2. Latest consensus state: current_term: 1 leader_uuid: "c67956db999547a5aef2a1a23f7b54e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c67956db999547a5aef2a1a23f7b54e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38193 } } }
I20260925 15:02:07.919977 29087 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:07.920001 29088 sys_catalog.cc:458] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:07.920182 29095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:07.920748 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:07.918430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34643 (local address 127.0.0.1:38193)
0925 15:02:07.919008 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:07.919018 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:07.919064 (+ 46us) server_negotiation.cc:407] Connection header received
0925 15:02:07.919189 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:07.919196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:07.919227 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:07.919695 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:07.920217 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:07.920221 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:07.920221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:07.920225 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:07.920266 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:07.920283 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:07.920351 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:07.920422 (+ 71us) server_negotiation.cc:299] Negotiation successful
0925 15:02:07.920547 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":493,"threads_started":1}
I20260925 15:02:07.921066 29095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:07.921639 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:07.922456 29095 catalog_manager.cc:1347] Generated new cluster ID: 486a1ad6bb794118bfa8f63b272442e7
I20260925 15:02:07.922495 29095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:07.924662 29085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:07.923577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42461 (local address 127.0.0.1:38193)
0925 15:02:07.923610 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:07.923612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:07.923644 (+ 32us) server_negotiation.cc:407] Connection header received
0925 15:02:07.923663 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:07.923664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:07.923668 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:07.923757 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:07.924437 (+ 680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:07.924438 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:07.924438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:07.924442 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:07.924458 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:07.924463 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:07.924526 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:07.924574 (+ 48us) server_negotiation.cc:299] Negotiation successful
0925 15:02:07.924586 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:02:07.938166 29095 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:07.938354 29095 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:07.946048 29095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c67956db999547a5aef2a1a23f7b54e2: Generated new TSK 0
I20260925 15:02:07.946269 29095 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:07.955109 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29013
2026-09-25T15:02:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:08 dist-test-slave-15hz krb5kdc[29112](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:08 dist-test-slave-15hz krb5kdc[29112](info): set up 2 sockets
Sep 25 15:02:08 dist-test-slave-15hz krb5kdc[29112](info): commencing operation
krb5kdc: starting...
W20260925 15:02:08.903434 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:08 dist-test-slave-15hz krb5kdc[29112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:09.018193 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37881
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37881
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:09.101039 29125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:09.101277 29125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:09.101301 29125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:09.102464 29125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:09.102500 29125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:09.102514 29125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:09.102524 29125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:09.102535 29125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:09.104158 29125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37881
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:09.104367 29125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:09.104548 29125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:09.109268 29130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:09.109272 29131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:09.109315 29133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:09.109452 29125 server_base.cc:1034] running on GCE node
I20260925 15:02:09.109710 29125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:09.109963 29125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:09.111119 29125 hybrid_clock.cc:648] HybridClock initialized: now 1790348529111104 us; error 35 us; skew 500 ppm
Sep 25 15:02:09 dist-test-slave-15hz krb5kdc[29112](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348529, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:09.118664 29125 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:09.119167 29125 webserver.cc:460] Webserver started at http://127.0.0.1:46577/ using document root <none> and password file <none>
I20260925 15:02:09.119395 29125 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:09.119442 29125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:09.119606 29125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:09.120738 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "46163dbe0b614ffe86e48a5daea0f268"
format_stamp: "Formatted at 2026-09-25 15:02:09 on dist-test-slave-15hz"
server_key: "1949d042dc11cf3eaddf5d48c275128e"
server_key_iv: "1537ce5ac20ade9b3ef7589cdd9a0dc9"
server_key_version: "encryptionkey@0"
I20260925 15:02:09.121073 29125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "46163dbe0b614ffe86e48a5daea0f268"
format_stamp: "Formatted at 2026-09-25 15:02:09 on dist-test-slave-15hz"
server_key: "1949d042dc11cf3eaddf5d48c275128e"
server_key_iv: "1537ce5ac20ade9b3ef7589cdd9a0dc9"
server_key_version: "encryptionkey@0"
I20260925 15:02:09.122565 29125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:02:09.123549 29140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:09.123930 29125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:09.124042 29125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "46163dbe0b614ffe86e48a5daea0f268"
format_stamp: "Formatted at 2026-09-25 15:02:09 on dist-test-slave-15hz"
server_key: "1949d042dc11cf3eaddf5d48c275128e"
server_key_iv: "1537ce5ac20ade9b3ef7589cdd9a0dc9"
server_key_version: "encryptionkey@0"
I20260925 15:02:09.124136 29125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:09.133948 29125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:09.136018 29125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:09.136147 29125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:09.139843 29125 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37881
I20260925 15:02:09.139871 29192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37881 every 8 connection(s)
I20260925 15:02:09.140214 29125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:09.141155 29193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:09.143371 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29125
I20260925 15:02:09.143496 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:09.143568 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: Bootstrap starting.
I20260925 15:02:09.143735 8247 external_mini_cluster.cc:1428] Setting key 3363fa68f63be51487f57762e85f38a4
I20260925 15:02:09.144269 29193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:09.144591 29193 log.cc:824] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:09.145309 29193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: No bootstrap required, opened a new log
I20260925 15:02:09.147477 29193 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260925 15:02:09.147673 29193 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:09.147701 29193 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46163dbe0b614ffe86e48a5daea0f268, State: Initialized, Role: FOLLOWER
I20260925 15:02:09.147823 29193 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260925 15:02:09.147886 29193 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:09.147908 29193 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:09.147951 29193 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:09.148615 29193 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260925 15:02:09.148725 29193 leader_election.cc:302] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46163dbe0b614ffe86e48a5daea0f268; no voters:
I20260925 15:02:09.148890 29193 leader_election.cc:288] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:09.148938 29198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:09.149121 29198 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [term 1 LEADER]: Becoming Leader. State: Replica: 46163dbe0b614ffe86e48a5daea0f268, State: Running, Role: LEADER
I20260925 15:02:09.149133 29193 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:09.149231 29198 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } }
I20260925 15:02:09.149577 29198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46163dbe0b614ffe86e48a5daea0f268. Latest consensus state: current_term: 1 leader_uuid: "46163dbe0b614ffe86e48a5daea0f268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } } }
I20260925 15:02:09.149590 29200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46163dbe0b614ffe86e48a5daea0f268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46163dbe0b614ffe86e48a5daea0f268" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37881 } } }
I20260925 15:02:09.149653 29198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:09.149683 29200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268 [sys.catalog]: This master's current role is: LEADER
W20260925 15:02:09.151000 29214 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:09.151054 29214 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:09.151095 29204 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:09.151260 29204 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 25 15:02:09 dist-test-slave-15hz krb5kdc[29112](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348528, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:02:09.152987 29204 catalog_manager.cc:1347] Generated new cluster ID: f5e3d025f0324f85b047034a4fbe1a57
I20260925 15:02:09.153040 29204 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:09.155110 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:09.144331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:49083 (local address 127.0.0.1:37881)
0925 15:02:09.144543 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:09.144553 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:09.144591 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:02:09.145120 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:09.145129 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:09.145197 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:09.145347 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:09.145945 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:09.147369 (+ 1424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:09.148316 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:09.148930 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:09.149057 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:09.151889 (+ 2832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:09.151893 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:09.151893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:09.151905 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:09.153565 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:09.153911 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:09.153914 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:09.153914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:09.153995 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:09.154190 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:09.154191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:09.154191 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:09.154459 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:09.154621 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:09.154758 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:02:09.154942 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260925 15:02:09.155972 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:09.162695 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:09.157183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:51787 (local address 127.0.0.1:37881)
0925 15:02:09.157218 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:09.157219 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:09.157318 (+ 99us) server_negotiation.cc:407] Connection header received
0925 15:02:09.157373 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:09.157373 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:09.157383 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:09.157433 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:09.157903 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:09.158397 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:09.159069 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:09.159288 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:09.159342 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:09.160454 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:09.160455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:09.160455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:09.160462 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:09.161673 (+ 1211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:09.162000 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:09.162001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:09.162002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:09.162057 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:09.162240 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:09.162241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:09.162241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:09.162344 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:09.162434 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:09.162547 (+ 113us) server_negotiation.cc:299] Negotiation successful
0925 15:02:09.162604 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:09.169620 29204 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:09.170356 29204 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:09.185966 29204 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46163dbe0b614ffe86e48a5daea0f268: Generated new TSK 0
I20260925 15:02:09.186203 29204 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:09.188719 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29125
W20260925 15:02:09.192772 29110 connection.cc:535] client connection to 127.0.0.1:37881 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37881): Connection reset by peer (error 104)
2026-09-25T15:02:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:09 dist-test-slave-15hz krb5kdc[29224](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:09 dist-test-slave-15hz krb5kdc[29224](info): set up 2 sockets
Sep 25 15:02:09 dist-test-slave-15hz krb5kdc[29224](info): commencing operation
krb5kdc: starting...
W20260925 15:02:10.139357 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:10 dist-test-slave-15hz krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348530, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:10.256278 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39383
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:10.337842 29237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:10.338080 29237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:10.338106 29237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:10.339390 29237 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:10.339460 29237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:10.339475 29237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:10.339486 29237 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:10.339498 29237 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:10.341540 29237 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29237
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:10.341830 29237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:10.342073 29237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:10.346868 29243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:10.346892 29242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:10.346873 29245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:10.347082 29237 server_base.cc:1034] running on GCE node
I20260925 15:02:10.347317 29237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:10.347647 29237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:10.348786 29237 hybrid_clock.cc:648] HybridClock initialized: now 1790348530348772 us; error 31 us; skew 500 ppm
Sep 25 15:02:10 dist-test-slave-15hz krb5kdc[29224](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348530, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:10.351506 29237 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:10.351995 29237 webserver.cc:460] Webserver started at http://127.0.0.1:42653/ using document root <none> and password file <none>
I20260925 15:02:10.352205 29237 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:10.352253 29237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:10.352378 29237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:10.353473 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "2f1b0f4b23a74248ae72481be990a995"
format_stamp: "Formatted at 2026-09-25 15:02:10 on dist-test-slave-15hz"
server_key: "40ca606bb3a8c42b980cc030327ffc72"
server_key_iv: "b1af10a22a498ad6c1fd1aa448d6af33"
server_key_version: "encryptionkey@0"
I20260925 15:02:10.353824 29237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "2f1b0f4b23a74248ae72481be990a995"
format_stamp: "Formatted at 2026-09-25 15:02:10 on dist-test-slave-15hz"
server_key: "40ca606bb3a8c42b980cc030327ffc72"
server_key_iv: "b1af10a22a498ad6c1fd1aa448d6af33"
server_key_version: "encryptionkey@0"
I20260925 15:02:10.355149 29237 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:10.355990 29252 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:10.356231 29237 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260925 15:02:10.356325 29237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "2f1b0f4b23a74248ae72481be990a995"
format_stamp: "Formatted at 2026-09-25 15:02:10 on dist-test-slave-15hz"
server_key: "40ca606bb3a8c42b980cc030327ffc72"
server_key_iv: "b1af10a22a498ad6c1fd1aa448d6af33"
server_key_version: "encryptionkey@0"
I20260925 15:02:10.356392 29237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:10.372381 29237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:10.374696 29237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:10.374841 29237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:10.379045 29237 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39383
I20260925 15:02:10.379076 29304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39383 every 8 connection(s)
I20260925 15:02:10.379426 29237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:10.380451 29305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:10.381131 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29237
I20260925 15:02:10.381245 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:10.381497 8247 external_mini_cluster.cc:1428] Setting key 6ae04a419982ee01b226ea1a1855d658
I20260925 15:02:10.383823 29305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995: Bootstrap starting.
I20260925 15:02:10.384580 29305 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:10.384876 29305 log.cc:824] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:10.385609 29305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995: No bootstrap required, opened a new log
I20260925 15:02:10.387806 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:10.382154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:33995 (local address 127.0.0.1:39383)
0925 15:02:10.382317 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:10.382328 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:10.382362 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:02:10.382902 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:10.382912 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:10.382985 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:10.383142 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:10.383382 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:10.384767 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:10.385585 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:10.386184 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:10.386331 (+ 147us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:10.387027 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:10.387033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:10.387033 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:10.387244 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:10.387287 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:10.387305 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:10.387404 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:10.387499 (+ 95us) server_negotiation.cc:299] Negotiation successful
0925 15:02:10.387576 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260925 15:02:10.387988 29305 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260925 15:02:10.388178 29305 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:10.388213 29305 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f1b0f4b23a74248ae72481be990a995, State: Initialized, Role: FOLLOWER
I20260925 15:02:10.388335 29305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260925 15:02:10.388422 29305 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:10.388456 29305 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:10.388504 29305 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:10.388846 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:10.389202 29305 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260925 15:02:10.389308 29305 leader_election.cc:302] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f1b0f4b23a74248ae72481be990a995; no voters:
I20260925 15:02:10.389501 29305 leader_election.cc:288] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:10.389562 29310 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:10.389761 29310 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [term 1 LEADER]: Becoming Leader. State: Replica: 2f1b0f4b23a74248ae72481be990a995, State: Running, Role: LEADER
I20260925 15:02:10.389779 29305 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:10.389863 29310 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } }
I20260925 15:02:10.390198 29310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f1b0f4b23a74248ae72481be990a995. Latest consensus state: current_term: 1 leader_uuid: "2f1b0f4b23a74248ae72481be990a995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } } }
I20260925 15:02:10.390276 29310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:10.390235 29311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f1b0f4b23a74248ae72481be990a995" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f1b0f4b23a74248ae72481be990a995" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39383 } } }
I20260925 15:02:10.390316 29311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:10.390463 29321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:10.390980 29321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:10.392581 29321 catalog_manager.cc:1347] Generated new cluster ID: a2c0f9bcaae244ef8c130565d27e724e
I20260925 15:02:10.392640 29321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:10.395607 29308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:10.391877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:35679 (local address 127.0.0.1:39383)
0925 15:02:10.391916 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:10.391918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:10.392052 (+ 134us) server_negotiation.cc:407] Connection header received
0925 15:02:10.392149 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:10.392150 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:10.392160 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:10.392215 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:10.392433 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:10.393174 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:10.394215 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:10.394548 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:10.394618 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:10.395326 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:10.395327 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:10.395327 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:10.395332 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:10.395349 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:10.395354 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:10.395416 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:10.395479 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:02:10.395493 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:02:10.423987 29321 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:10.424782 29321 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:10.431326 29321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f1b0f4b23a74248ae72481be990a995: Generated new TSK 0
I20260925 15:02:10.431603 29321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:10.434254 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29237
W20260925 15:02:10.438396 29222 connection.cc:535] client connection to 127.0.0.1:39383 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39383): Connection reset by peer (error 104)
2026-09-25T15:02:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1245 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:10 dist-test-slave-15hz krb5kdc[29336](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:10 dist-test-slave-15hz krb5kdc[29336](info): set up 2 sockets
Sep 25 15:02:10 dist-test-slave-15hz krb5kdc[29336](info): commencing operation
krb5kdc: starting...
W20260925 15:02:11.387375 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:11 dist-test-slave-15hz krb5kdc[29336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348531, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:11.504294 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33853
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:11.584944 29349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:11.585184 29349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:11.585210 29349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:11.586469 29349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:11.586519 29349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:11.586534 29349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:11.586544 29349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:11.586555 29349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:11.588162 29349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29349
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:11.588405 29349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:11.588636 29349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:11.593382 29354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:11.593400 29355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:11.593452 29357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:11.593590 29349 server_base.cc:1034] running on GCE node
I20260925 15:02:11.593816 29349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:11.594086 29349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:11.595233 29349 hybrid_clock.cc:648] HybridClock initialized: now 1790348531595220 us; error 28 us; skew 500 ppm
Sep 25 15:02:11 dist-test-slave-15hz krb5kdc[29336](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348531, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:11.597800 29349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:11.598302 29349 webserver.cc:460] Webserver started at http://127.0.0.1:40907/ using document root <none> and password file <none>
I20260925 15:02:11.598510 29349 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:11.598562 29349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:11.598678 29349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:11.599777 29349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "c6dcf79ef01842aeab0b6af8f1211c6d"
format_stamp: "Formatted at 2026-09-25 15:02:11 on dist-test-slave-15hz"
server_key: "eb96166ad7c9ce47baf2e67ab76fcb65"
server_key_iv: "5f05019a77f6c04b8994ea045fdc3a77"
server_key_version: "encryptionkey@0"
I20260925 15:02:11.600116 29349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "c6dcf79ef01842aeab0b6af8f1211c6d"
format_stamp: "Formatted at 2026-09-25 15:02:11 on dist-test-slave-15hz"
server_key: "eb96166ad7c9ce47baf2e67ab76fcb65"
server_key_iv: "5f05019a77f6c04b8994ea045fdc3a77"
server_key_version: "encryptionkey@0"
I20260925 15:02:11.601348 29349 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:11.602011 29364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:11.602231 29349 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:11.602324 29349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "c6dcf79ef01842aeab0b6af8f1211c6d"
format_stamp: "Formatted at 2026-09-25 15:02:11 on dist-test-slave-15hz"
server_key: "eb96166ad7c9ce47baf2e67ab76fcb65"
server_key_iv: "5f05019a77f6c04b8994ea045fdc3a77"
server_key_version: "encryptionkey@0"
I20260925 15:02:11.602392 29349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:11.605795 29349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:11.607822 29349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:11.607956 29349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:11.611860 29349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33853
I20260925 15:02:11.611897 29416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33853 every 8 connection(s)
I20260925 15:02:11.612223 29349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:11.613157 29417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:11.615481 29417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d: Bootstrap starting.
I20260925 15:02:11.616060 29417 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:11.616297 29417 log.cc:824] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:11.616928 29417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d: No bootstrap required, opened a new log
I20260925 15:02:11.618462 29417 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 15:02:11.618603 29417 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:11.618624 29417 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dcf79ef01842aeab0b6af8f1211c6d, State: Initialized, Role: FOLLOWER
I20260925 15:02:11.618733 29417 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 15:02:11.618808 29417 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:11.618831 29417 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:11.618865 29417 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:11.619000 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29349
I20260925 15:02:11.619112 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:11.619287 8247 external_mini_cluster.cc:1428] Setting key c1bc3c40fde3e46d90d8cc509d45e14f
I20260925 15:02:11.619473 29417 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 15:02:11.619608 29417 leader_election.cc:302] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6dcf79ef01842aeab0b6af8f1211c6d; no voters:
I20260925 15:02:11.619781 29417 leader_election.cc:288] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:11.619825 29420 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:11.619954 29420 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [term 1 LEADER]: Becoming Leader. State: Replica: c6dcf79ef01842aeab0b6af8f1211c6d, State: Running, Role: LEADER
I20260925 15:02:11.620031 29417 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:11.620071 29420 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } }
I20260925 15:02:11.620456 29423 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260925 15:02:11.620581 29423 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:11.620986 29424 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6dcf79ef01842aeab0b6af8f1211c6d. Latest consensus state: current_term: 1 leader_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6dcf79ef01842aeab0b6af8f1211c6d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33853 } } }
I20260925 15:02:11.621009 29428 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:11.621112 29424 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:11.621577 29428 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:11.622491 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:11.619746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:43579 (local address 127.0.0.1:33853)
0925 15:02:11.619919 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:11.619929 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:11.620647 (+ 718us) server_negotiation.cc:407] Connection header received
0925 15:02:11.620772 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:11.620781 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:11.620815 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:11.620951 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:11.621958 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:11.621962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:11.621962 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:11.621966 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:11.622009 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:11.622024 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:11.622095 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:11.622173 (+ 78us) server_negotiation.cc:299] Negotiation successful
0925 15:02:11.622217 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":78,"threads_started":1}
I20260925 15:02:11.623351 29428 catalog_manager.cc:1347] Generated new cluster ID: 2f33cfdc5ec048bfb69336a69747b4a3
I20260925 15:02:11.623402 29428 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:11.623387 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:11.626000 29422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:11.624933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:59827 (local address 127.0.0.1:33853)
0925 15:02:11.624967 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:11.624969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:11.625071 (+ 102us) server_negotiation.cc:407] Connection header received
0925 15:02:11.625127 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:11.625127 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:11.625150 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:11.625205 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:11.625797 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:11.625798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:11.625798 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:11.625803 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:11.625819 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:11.625823 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:11.625881 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:11.625920 (+ 39us) server_negotiation.cc:299] Negotiation successful
0925 15:02:11.625934 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:11.632339 29428 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:11.632506 29428 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:11.638804 29428 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6dcf79ef01842aeab0b6af8f1211c6d: Generated new TSK 0
I20260925 15:02:11.639035 29428 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:11.644587 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29349
2026-09-25T15:02:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1210 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:11 dist-test-slave-15hz krb5kdc[29448](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:11 dist-test-slave-15hz krb5kdc[29448](info): set up 2 sockets
Sep 25 15:02:11 dist-test-slave-15hz krb5kdc[29448](info): commencing operation
krb5kdc: starting...
W20260925 15:02:12.595408 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:12.706359 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37005
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37005
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:12.786403 29461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:12.786636 29461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:12.786669 29461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:12.787957 29461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:12.788008 29461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:12.788028 29461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:12.788043 29461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:12.788057 29461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:12.789623 29461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37005
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37005
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:12.789892 29461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:12.790117 29461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:12.794799 29467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:12.794780 29469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:12.794780 29466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:12.794813 29461 server_base.cc:1034] running on GCE node
I20260925 15:02:12.795240 29461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:12.795558 29461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:12.796728 29461 hybrid_clock.cc:648] HybridClock initialized: now 1790348532796710 us; error 30 us; skew 500 ppm
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29448](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348532, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:12.799283 29461 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:12.799789 29461 webserver.cc:460] Webserver started at http://127.0.0.1:42751/ using document root <none> and password file <none>
I20260925 15:02:12.800021 29461 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:12.800076 29461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:12.800184 29461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:12.801251 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "6404ecf855ab4033a0eb1402af88d90d"
format_stamp: "Formatted at 2026-09-25 15:02:12 on dist-test-slave-15hz"
server_key: "5e8a131ff5bfd27bdfe0b11e8d3bb955"
server_key_iv: "531aaa88a909ed24a4e0c449c5652e28"
server_key_version: "encryptionkey@0"
I20260925 15:02:12.801575 29461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "6404ecf855ab4033a0eb1402af88d90d"
format_stamp: "Formatted at 2026-09-25 15:02:12 on dist-test-slave-15hz"
server_key: "5e8a131ff5bfd27bdfe0b11e8d3bb955"
server_key_iv: "531aaa88a909ed24a4e0c449c5652e28"
server_key_version: "encryptionkey@0"
I20260925 15:02:12.802908 29461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:12.803663 29476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:12.803817 29461 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:12.803902 29461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "6404ecf855ab4033a0eb1402af88d90d"
format_stamp: "Formatted at 2026-09-25 15:02:12 on dist-test-slave-15hz"
server_key: "5e8a131ff5bfd27bdfe0b11e8d3bb955"
server_key_iv: "531aaa88a909ed24a4e0c449c5652e28"
server_key_version: "encryptionkey@0"
I20260925 15:02:12.804020 29461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:12.817914 29461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:12.820109 29461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:12.820225 29461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:12.824003 29461 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37005
I20260925 15:02:12.824048 29528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37005 every 8 connection(s)
I20260925 15:02:12.824374 29461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:12.825281 29529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:12.827509 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d: Bootstrap starting.
I20260925 15:02:12.828148 29529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:12.828409 29529 log.cc:824] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:12.829077 29529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d: No bootstrap required, opened a new log
I20260925 15:02:12.830641 29529 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260925 15:02:12.830823 29529 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:12.830859 29529 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6404ecf855ab4033a0eb1402af88d90d, State: Initialized, Role: FOLLOWER
I20260925 15:02:12.830869 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29461
I20260925 15:02:12.830962 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:12.831017 29529 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260925 15:02:12.831095 29529 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:12.831139 29529 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:12.831161 8247 external_mini_cluster.cc:1428] Setting key 74a03935df95f851f5ca9b34a711937f
I20260925 15:02:12.831185 29529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:12.831822 29529 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260925 15:02:12.831929 29529 leader_election.cc:302] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6404ecf855ab4033a0eb1402af88d90d; no voters:
I20260925 15:02:12.832085 29529 leader_election.cc:288] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:12.832106 29534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:12.832212 29534 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [term 1 LEADER]: Becoming Leader. State: Replica: 6404ecf855ab4033a0eb1402af88d90d, State: Running, Role: LEADER
I20260925 15:02:12.832293 29534 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } }
I20260925 15:02:12.832355 29529 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:12.832607 29536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6404ecf855ab4033a0eb1402af88d90d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } } }
I20260925 15:02:12.832695 29536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:12.832831 29535 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6404ecf855ab4033a0eb1402af88d90d. Latest consensus state: current_term: 1 leader_uuid: "6404ecf855ab4033a0eb1402af88d90d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6404ecf855ab4033a0eb1402af88d90d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37005 } } }
I20260925 15:02:12.832888 29535 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:12.832991 29539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:12.833530 29539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:12.836836 29539 catalog_manager.cc:1347] Generated new cluster ID: eaf6fd09ad99433e9c616f19ad1d5885
I20260925 15:02:12.836891 29539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29448](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348532, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260925 15:02:12.842105 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:12.831938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:56901 (local address 127.0.0.1:37005)
0925 15:02:12.832095 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:12.832105 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:12.832142 (+ 37us) server_negotiation.cc:407] Connection header received
0925 15:02:12.832265 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:12.832273 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:12.832360 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:12.833321 (+ 961us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:12.833420 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:12.834809 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:12.835858 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:12.836296 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:12.836376 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:12.838728 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:12.838732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:12.838732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:12.838743 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:12.840664 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:12.841027 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:12.841030 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:12.841030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:12.841122 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:12.841316 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:12.841316 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:12.841317 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:12.841550 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:12.841693 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:12.841813 (+ 120us) server_negotiation.cc:299] Negotiation successful
0925 15:02:12.841909 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":80,"threads_started":1}
I20260925 15:02:12.842940 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:12.844177 29539 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:12.844863 29539 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:12.849895 29533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:12.844284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:45889 (local address 127.0.0.1:37005)
0925 15:02:12.844318 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:12.844320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:12.844530 (+ 210us) server_negotiation.cc:407] Connection header received
0925 15:02:12.844583 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:12.844584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:12.844604 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:12.844646 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:12.845148 (+ 502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:12.845636 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:12.846298 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:12.846532 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:12.846584 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:12.847643 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:12.847644 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:12.847645 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:12.847651 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:12.848889 (+ 1238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:12.849215 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:12.849216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:12.849216 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:12.849267 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:12.849448 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:12.849449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:12.849449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:12.849558 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:12.849644 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:12.849755 (+ 111us) server_negotiation.cc:299] Negotiation successful
0925 15:02:12.849811 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:02:12.853325 29539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6404ecf855ab4033a0eb1402af88d90d: Generated new TSK 0
I20260925 15:02:12.853523 29539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:12.857492 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29461
W20260925 15:02:12.861379 29446 connection.cc:535] client connection to 127.0.0.1:37005 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37005): Connection reset by peer (error 104)
2026-09-25T15:02:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1212 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29560](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29560](info): set up 2 sockets
Sep 25 15:02:12 dist-test-slave-15hz krb5kdc[29560](info): commencing operation
krb5kdc: starting...
W20260925 15:02:13.795428 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:13 dist-test-slave-15hz krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:13.915318 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39183
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34485
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:13.994287 29573 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:13.994517 29573 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:13.994550 29573 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:13.995784 29573 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:13.995833 29573 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:13.995852 29573 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:13.995867 29573 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:13.995882 29573 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:13.997445 29573 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34485
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39183
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29573
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:13.997732 29573 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:13.997957 29573 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:14.002996 29579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:14.002916 29578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:14.002923 29581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:14.003139 29573 server_base.cc:1034] running on GCE node
I20260925 15:02:14.003440 29573 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:14.003753 29573 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:14.004899 29573 hybrid_clock.cc:648] HybridClock initialized: now 1790348534004871 us; error 36 us; skew 500 ppm
Sep 25 15:02:14 dist-test-slave-15hz krb5kdc[29560](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348534, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:14.007249 29573 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:14.007725 29573 webserver.cc:460] Webserver started at http://127.0.0.1:38461/ using document root <none> and password file <none>
I20260925 15:02:14.007937 29573 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:14.007985 29573 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:14.008112 29573 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:14.009310 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "b7065bcee2fb434793050f1563d35674"
format_stamp: "Formatted at 2026-09-25 15:02:14 on dist-test-slave-15hz"
server_key: "3f9061a6e6227815d6fc6b74bca8befe"
server_key_iv: "a8f7cf551ad46c4e8cb08b1405d184de"
server_key_version: "encryptionkey@0"
I20260925 15:02:14.009693 29573 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "b7065bcee2fb434793050f1563d35674"
format_stamp: "Formatted at 2026-09-25 15:02:14 on dist-test-slave-15hz"
server_key: "3f9061a6e6227815d6fc6b74bca8befe"
server_key_iv: "a8f7cf551ad46c4e8cb08b1405d184de"
server_key_version: "encryptionkey@0"
I20260925 15:02:14.010916 29573 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:14.011698 29588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:14.011847 29573 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:14.011930 29573 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "b7065bcee2fb434793050f1563d35674"
format_stamp: "Formatted at 2026-09-25 15:02:14 on dist-test-slave-15hz"
server_key: "3f9061a6e6227815d6fc6b74bca8befe"
server_key_iv: "a8f7cf551ad46c4e8cb08b1405d184de"
server_key_version: "encryptionkey@0"
I20260925 15:02:14.011999 29573 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:14.023156 29573 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:14.025187 29573 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:14.025329 29573 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:14.029181 29573 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39183
I20260925 15:02:14.029258 29640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39183 every 8 connection(s)
I20260925 15:02:14.029565 29573 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:14.029846 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29573
I20260925 15:02:14.029985 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:14.030220 8247 external_mini_cluster.cc:1428] Setting key 15ba4b8ccc08523ffcd6415e968294d4
I20260925 15:02:14.030550 29641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:14.033632 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: Bootstrap starting.
I20260925 15:02:14.034325 29641 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:14.034586 29641 log.cc:824] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:14.035286 29641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: No bootstrap required, opened a new log
I20260925 15:02:14.036185 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:14.030723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:48733 (local address 127.0.0.1:39183)
0925 15:02:14.030888 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:14.030897 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:14.030933 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:02:14.031680 (+ 747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:14.031689 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:14.031745 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:14.031863 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:14.032141 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:14.033071 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:14.034154 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:14.034618 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:14.034732 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:14.035401 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:14.035407 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:14.035407 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:14.035656 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:14.035694 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:14.035708 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:14.035800 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:14.035927 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:02:14.035955 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
I20260925 15:02:14.037297 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:14.037374 29641 raft_consensus.cc:348] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 15:02:14.037515 29641 raft_consensus.cc:374] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:14.037537 29641 raft_consensus.cc:729] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b7065bcee2fb434793050f1563d35674, State: Initialized, Role: FOLLOWER
I20260925 15:02:14.037675 29641 consensus_queue.cc:260] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 15:02:14.037741 29641 raft_consensus.cc:388] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:14.037771 29641 raft_consensus.cc:482] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:14.037818 29641 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:14.038417 29641 raft_consensus.cc:504] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 15:02:14.038517 29641 leader_election.cc:302] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b7065bcee2fb434793050f1563d35674; no voters:
I20260925 15:02:14.038689 29641 leader_election.cc:288] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:14.038733 29650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:14.038882 29650 raft_consensus.cc:686] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [term 1 LEADER]: Becoming Leader. State: Replica: b7065bcee2fb434793050f1563d35674, State: Running, Role: LEADER
I20260925 15:02:14.039011 29641 sys_catalog.cc:564] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:14.039043 29650 consensus_queue.cc:237] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } }
I20260925 15:02:14.039515 29653 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b7065bcee2fb434793050f1563d35674. Latest consensus state: current_term: 1 leader_uuid: "b7065bcee2fb434793050f1563d35674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } } }
I20260925 15:02:14.039649 29653 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:14.039857 29652 sys_catalog.cc:455] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b7065bcee2fb434793050f1563d35674" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b7065bcee2fb434793050f1563d35674" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39183 } } }
I20260925 15:02:14.039960 29652 sys_catalog.cc:458] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674 [sys.catalog]: This master's current role is: LEADER
W20260925 15:02:14.040766 29667 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:14.041043 29667 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:14.041147 29660 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:14.041313 29660 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:14.042092 29644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:14.038748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:54447 (local address 127.0.0.1:39183)
0925 15:02:14.038795 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:14.038798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:14.038946 (+ 148us) server_negotiation.cc:407] Connection header received
0925 15:02:14.039039 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:14.039040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:14.039052 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:14.039122 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:14.039295 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:14.040002 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:14.040785 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:14.041050 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:14.041114 (+ 64us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:14.041826 (+ 712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:14.041828 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:14.041828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:14.041833 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:14.041850 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:14.041855 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:14.041906 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:14.041999 (+ 93us) server_negotiation.cc:299] Negotiation successful
0925 15:02:14.042010 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:02:14.042825 29660 catalog_manager.cc:1347] Generated new cluster ID: c47aa8b551624214917691d73fecbac6
I20260925 15:02:14.042876 29660 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:14.052244 29660 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:14.052812 29660 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:14.059026 29660 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b7065bcee2fb434793050f1563d35674: Generated new TSK 0
I20260925 15:02:14.059231 29660 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:14.068542 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29573
W20260925 15:02:14.072569 29558 connection.cc:535] client connection to 127.0.0.1:39183 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39183): Connection reset by peer (error 104)
2026-09-25T15:02:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:14 dist-test-slave-15hz krb5kdc[29672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:14 dist-test-slave-15hz krb5kdc[29672](info): set up 2 sockets
Sep 25 15:02:14 dist-test-slave-15hz krb5kdc[29672](info): commencing operation
krb5kdc: starting...
W20260925 15:02:15.019490 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:15 dist-test-slave-15hz krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348535, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260925 15:02:15.137648 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43401
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43401
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:15.217737 29685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:15.217967 29685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:15.217999 29685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:15.219169 29685 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:15.219209 29685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:15.219221 29685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:15.219233 29685 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:15.219244 29685 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:15.220794 29685 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43401
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:15.221026 29685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:15.221206 29685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:15.226094 29690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:15.226105 29693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:15.226104 29691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:15.226182 29685 server_base.cc:1034] running on GCE node
I20260925 15:02:15.226504 29685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:15.226843 29685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:15.228047 29685 hybrid_clock.cc:648] HybridClock initialized: now 1790348535228032 us; error 34 us; skew 500 ppm
Sep 25 15:02:15 dist-test-slave-15hz krb5kdc[29672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348535, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:15.230620 29685 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260925 15:02:15.231117 29685 webserver.cc:460] Webserver started at http://127.0.0.1:42681/ using document root <none> and password file <none>
I20260925 15:02:15.231333 29685 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:15.231382 29685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:15.231513 29685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:15.232676 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "7ed47d15768d409d92404b4eae16fa61"
format_stamp: "Formatted at 2026-09-25 15:02:15 on dist-test-slave-15hz"
server_key: "18cf6e73dc4af2a3eca8478433e6f34d"
server_key_iv: "221354396497e56850629e6a3845b57d"
server_key_version: "encryptionkey@0"
I20260925 15:02:15.233027 29685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "7ed47d15768d409d92404b4eae16fa61"
format_stamp: "Formatted at 2026-09-25 15:02:15 on dist-test-slave-15hz"
server_key: "18cf6e73dc4af2a3eca8478433e6f34d"
server_key_iv: "221354396497e56850629e6a3845b57d"
server_key_version: "encryptionkey@0"
I20260925 15:02:15.234382 29685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:15.235255 29700 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:15.235441 29685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:15.235569 29685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "7ed47d15768d409d92404b4eae16fa61"
format_stamp: "Formatted at 2026-09-25 15:02:15 on dist-test-slave-15hz"
server_key: "18cf6e73dc4af2a3eca8478433e6f34d"
server_key_iv: "221354396497e56850629e6a3845b57d"
server_key_version: "encryptionkey@0"
I20260925 15:02:15.235642 29685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:15.239071 29685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:15.241355 29685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:15.241495 29685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:15.245478 29685 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43401
I20260925 15:02:15.245525 29752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43401 every 8 connection(s)
I20260925 15:02:15.245843 29685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:15.246840 29753 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:15.249215 29753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61: Bootstrap starting.
I20260925 15:02:15.249785 29753 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:15.250056 29753 log.cc:824] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:15.250759 29753 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61: No bootstrap required, opened a new log
I20260925 15:02:15.252519 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29685
I20260925 15:02:15.252637 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:15.252488 29753 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } }
I20260925 15:02:15.252681 29753 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:15.252720 29753 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ed47d15768d409d92404b4eae16fa61, State: Initialized, Role: FOLLOWER
I20260925 15:02:15.252854 8247 external_mini_cluster.cc:1428] Setting key 32e54459f660d889c6826dae19ccd967
I20260925 15:02:15.252874 29753 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } }
I20260925 15:02:15.252954 29753 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:15.253002 29753 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:15.253044 29753 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:15.253710 29753 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } }
I20260925 15:02:15.253808 29753 leader_election.cc:302] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ed47d15768d409d92404b4eae16fa61; no voters:
I20260925 15:02:15.253962 29753 leader_election.cc:288] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:15.253990 29758 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:15.254148 29758 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [term 1 LEADER]: Becoming Leader. State: Replica: 7ed47d15768d409d92404b4eae16fa61, State: Running, Role: LEADER
I20260925 15:02:15.254211 29753 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:15.254242 29758 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } }
I20260925 15:02:15.254595 29758 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ed47d15768d409d92404b4eae16fa61. Latest consensus state: current_term: 1 leader_uuid: "7ed47d15768d409d92404b4eae16fa61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } } }
I20260925 15:02:15.254665 29758 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:15.254807 29759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ed47d15768d409d92404b4eae16fa61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ed47d15768d409d92404b4eae16fa61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43401 } } }
I20260925 15:02:15.254874 29759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:15.254989 29765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:15.255511 29765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:15.255574 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:15.253363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:37961 (local address 127.0.0.1:43401)
0925 15:02:15.253838 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:15.253848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:15.253883 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:02:15.254018 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:15.254026 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:15.254058 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:15.254489 (+ 431us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:15.254971 (+ 482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:15.254975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:15.254975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:15.254979 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:15.255023 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:15.255038 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:15.255299 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:15.255331 (+ 32us) server_negotiation.cc:299] Negotiation successful
0925 15:02:15.255374 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":375,"threads_started":1}
I20260925 15:02:15.256488 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:15.257078 29765 catalog_manager.cc:1347] Generated new cluster ID: 02bc0326b4fb45deb9135e72bf8cebba
I20260925 15:02:15.257130 29765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:15.258786 29757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:15.257844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.19:46351 (local address 127.0.0.1:43401)
0925 15:02:15.257899 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:15.257901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:15.257990 (+ 89us) server_negotiation.cc:407] Connection header received
0925 15:02:15.258057 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:15.258057 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:15.258062 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:15.258105 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:15.258607 (+ 502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:15.258608 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:15.258608 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:15.258611 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:15.258624 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:15.258627 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:15.258668 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:15.258713 (+ 45us) server_negotiation.cc:299] Negotiation successful
0925 15:02:15.258725 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:02:15.267632 29765 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:15.267853 29765 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:15.273510 29765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ed47d15768d409d92404b4eae16fa61: Generated new TSK 0
I20260925 15:02:15.273741 29765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:15.282133 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29685
2026-09-25T15:02:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1213 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:15 dist-test-slave-15hz krb5kdc[29784](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:15 dist-test-slave-15hz krb5kdc[29784](info): set up 2 sockets
Sep 25 15:02:15 dist-test-slave-15hz krb5kdc[29784](info): commencing operation
krb5kdc: starting...
W20260925 15:02:16.235405 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:16.354772 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:35711
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:46389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:35711
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:16.435860 29797 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:16.436094 29797 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:16.436129 29797 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:16.437364 29797 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:16.437412 29797 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:16.437429 29797 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:16.437451 29797 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:16.437467 29797 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:16.439024 29797 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:46389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:35711
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:35711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29797
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:16.439286 29797 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:16.439509 29797 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:16.444424 29802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:16.444424 29803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:16.444438 29805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:16.444765 29797 server_base.cc:1034] running on GCE node
I20260925 15:02:16.445005 29797 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:16.445317 29797 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:16.446496 29797 hybrid_clock.cc:648] HybridClock initialized: now 1790348536446483 us; error 34 us; skew 500 ppm
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29784](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348536, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:16.449239 29797 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:16.449757 29797 webserver.cc:460] Webserver started at http://127.8.13.254:32979/ using document root <none> and password file <none>
I20260925 15:02:16.450001 29797 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:16.450045 29797 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:16.450139 29797 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:16.451256 29797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "f9b51270139e4c37b511a1fa86fb9f48"
format_stamp: "Formatted at 2026-09-25 15:02:16 on dist-test-slave-15hz"
server_key: "2095204f302fe72240a594261e06dc0d"
server_key_iv: "026712a5fdd3880d64eeeb4bf5e39967"
server_key_version: "encryptionkey@0"
I20260925 15:02:16.451624 29797 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "f9b51270139e4c37b511a1fa86fb9f48"
format_stamp: "Formatted at 2026-09-25 15:02:16 on dist-test-slave-15hz"
server_key: "2095204f302fe72240a594261e06dc0d"
server_key_iv: "026712a5fdd3880d64eeeb4bf5e39967"
server_key_version: "encryptionkey@0"
I20260925 15:02:16.452893 29797 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:02:16.453568 29812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:16.453733 29797 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:02:16.453820 29797 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "f9b51270139e4c37b511a1fa86fb9f48"
format_stamp: "Formatted at 2026-09-25 15:02:16 on dist-test-slave-15hz"
server_key: "2095204f302fe72240a594261e06dc0d"
server_key_iv: "026712a5fdd3880d64eeeb4bf5e39967"
server_key_version: "encryptionkey@0"
I20260925 15:02:16.453881 29797 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:16.464342 29797 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:16.466562 29797 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:16.466714 29797 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:16.470957 29797 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:35711
I20260925 15:02:16.470979 29864 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:35711 every 8 connection(s)
I20260925 15:02:16.471333 29797 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:16.472244 29865 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:16.474519 29865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48: Bootstrap starting.
I20260925 15:02:16.475075 29865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:16.475302 29865 log.cc:824] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:16.476076 29865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48: No bootstrap required, opened a new log
I20260925 15:02:16.477689 29865 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } }
I20260925 15:02:16.477838 29865 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:16.477859 29865 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b51270139e4c37b511a1fa86fb9f48, State: Initialized, Role: FOLLOWER
I20260925 15:02:16.477980 29865 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } }
I20260925 15:02:16.478055 29865 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:16.478087 29865 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:16.478117 29865 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:16.478940 29865 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } }
I20260925 15:02:16.479032 29865 leader_election.cc:302] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b51270139e4c37b511a1fa86fb9f48; no voters:
I20260925 15:02:16.479172 29865 leader_election.cc:288] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:16.479226 29868 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:16.479414 29865 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:16.479413 29868 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [term 1 LEADER]: Becoming Leader. State: Replica: f9b51270139e4c37b511a1fa86fb9f48, State: Running, Role: LEADER
I20260925 15:02:16.479626 29868 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } }
I20260925 15:02:16.480024 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29797
I20260925 15:02:16.479996 29869 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9b51270139e4c37b511a1fa86fb9f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } } }
I20260925 15:02:16.480073 29869 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:16.480145 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:16.480233 29870 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9b51270139e4c37b511a1fa86fb9f48. Latest consensus state: current_term: 1 leader_uuid: "f9b51270139e4c37b511a1fa86fb9f48" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b51270139e4c37b511a1fa86fb9f48" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 35711 } } }
I20260925 15:02:16.480278 29870 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:16.480341 8247 external_mini_cluster.cc:1428] Setting key 0abf0a651a05cd086a8fbe0c342cf627
I20260925 15:02:16.480624 29877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:16.481307 29877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:16.483619 29877 catalog_manager.cc:1347] Generated new cluster ID: d7b1cf320c654862a6a7bff8a90d9631
I20260925 15:02:16.483695 29877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29784](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:16.490859 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:16.480928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34923 (local address 127.8.13.254:35711)
0925 15:02:16.481089 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:16.481097 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:16.481122 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:02:16.481231 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:16.481237 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:16.481282 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:16.481433 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:16.482042 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:16.483445 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:16.484330 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:16.484771 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:16.484881 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:16.487144 (+ 2263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:16.487149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:16.487149 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:16.487161 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:16.489302 (+ 2141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:16.489697 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:16.489700 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:16.489700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:16.489805 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:16.490019 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:16.490020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:16.490020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:16.490286 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:16.490438 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:16.490565 (+ 127us) server_negotiation.cc:299] Negotiation successful
0925 15:02:16.490672 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":72,"threads_started":1}
I20260925 15:02:16.491586 29877 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:16.491959 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:16.492292 29877 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:16.496342 29877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9b51270139e4c37b511a1fa86fb9f48: Generated new TSK 0
I20260925 15:02:16.496562 29877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:16.505611 29886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:16.496083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46289 (local address 127.8.13.254:35711)
0925 15:02:16.496127 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:16.496129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:16.499281 (+ 3152us) server_negotiation.cc:407] Connection header received
0925 15:02:16.499326 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:16.499327 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:16.499339 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:16.499388 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:16.500029 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:16.500764 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:16.501505 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:16.501730 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:16.501785 (+ 55us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:16.503040 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:16.503041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:16.503042 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:16.503049 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:16.504469 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:16.504864 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:16.504865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:16.504866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:16.504919 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:16.505116 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:16.505117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:16.505118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:16.505223 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:16.505315 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:16.505455 (+ 140us) server_negotiation.cc:299] Negotiation successful
0925 15:02:16.505517 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:02:16.507946 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29797
W20260925 15:02:16.511713 29782 connection.cc:535] client connection to 127.8.13.254:35711 recv error: Network error: failed to read from TLS socket (remote: 127.8.13.254:35711): Connection reset by peer (error 104)
2026-09-25T15:02:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1225 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29896](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29896](info): set up 2 sockets
Sep 25 15:02:16 dist-test-slave-15hz krb5kdc[29896](info): commencing operation
krb5kdc: starting...
W20260925 15:02:17.451439 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:17 dist-test-slave-15hz krb5kdc[29896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:17.564723 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44027
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:37049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:44027
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:17.644315 29909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:17.644531 29909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:17.644554 29909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:17.645745 29909 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:17.645795 29909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:17.645808 29909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:17.645820 29909 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:17.645831 29909 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:17.647364 29909 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:37049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:44027
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:44027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29909
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:17.647619 29909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:17.647847 29909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:17.652523 29915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:17.652649 29917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:17.652578 29914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:17.652597 29909 server_base.cc:1034] running on GCE node
I20260925 15:02:17.652997 29909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:17.653282 29909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:17.654443 29909 hybrid_clock.cc:648] HybridClock initialized: now 1790348537654431 us; error 34 us; skew 500 ppm
Sep 25 15:02:17 dist-test-slave-15hz krb5kdc[29896](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348537, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:17.656891 29909 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:17.657326 29909 webserver.cc:460] Webserver started at http://127.8.13.254:37267/ using document root <none> and password file <none>
I20260925 15:02:17.657545 29909 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:17.657603 29909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:17.657728 29909 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:17.658844 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "f536630d0793400692a3a74a37981f1e"
format_stamp: "Formatted at 2026-09-25 15:02:17 on dist-test-slave-15hz"
server_key: "32c83ffea0fa54aae403f23b7959aab1"
server_key_iv: "1b8c6fb6da2b80b04319b1d99c758ba1"
server_key_version: "encryptionkey@0"
I20260925 15:02:17.659210 29909 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "f536630d0793400692a3a74a37981f1e"
format_stamp: "Formatted at 2026-09-25 15:02:17 on dist-test-slave-15hz"
server_key: "32c83ffea0fa54aae403f23b7959aab1"
server_key_iv: "1b8c6fb6da2b80b04319b1d99c758ba1"
server_key_version: "encryptionkey@0"
I20260925 15:02:17.660564 29909 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260925 15:02:17.661305 29924 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:17.661481 29909 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:17.661594 29909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "f536630d0793400692a3a74a37981f1e"
format_stamp: "Formatted at 2026-09-25 15:02:17 on dist-test-slave-15hz"
server_key: "32c83ffea0fa54aae403f23b7959aab1"
server_key_iv: "1b8c6fb6da2b80b04319b1d99c758ba1"
server_key_version: "encryptionkey@0"
I20260925 15:02:17.661746 29909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:17.678578 29909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:17.680670 29909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:17.680815 29909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:17.684792 29909 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:44027
I20260925 15:02:17.684837 29976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:44027 every 8 connection(s)
I20260925 15:02:17.685199 29909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:17.686195 29977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:17.688655 29977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e: Bootstrap starting.
I20260925 15:02:17.689280 29977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:17.689553 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 29909
I20260925 15:02:17.689584 29977 log.cc:824] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:17.689675 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:17.689857 8247 external_mini_cluster.cc:1428] Setting key 18e215d48ad07e80ce29d8115373809b
I20260925 15:02:17.690449 29977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e: No bootstrap required, opened a new log
I20260925 15:02:17.693347 29977 raft_consensus.cc:348] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } }
I20260925 15:02:17.693522 29977 raft_consensus.cc:374] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:17.693547 29977 raft_consensus.cc:729] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f536630d0793400692a3a74a37981f1e, State: Initialized, Role: FOLLOWER
I20260925 15:02:17.693666 29977 consensus_queue.cc:260] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } }
I20260925 15:02:17.693739 29977 raft_consensus.cc:388] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:17.693764 29977 raft_consensus.cc:482] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:17.693812 29977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:17.694619 29977 raft_consensus.cc:504] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } }
I20260925 15:02:17.694739 29977 leader_election.cc:302] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f536630d0793400692a3a74a37981f1e; no voters:
I20260925 15:02:17.694960 29977 leader_election.cc:288] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:17.695034 29982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:17.695236 29977 sys_catalog.cc:564] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:17.695261 29982 raft_consensus.cc:686] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [term 1 LEADER]: Becoming Leader. State: Replica: f536630d0793400692a3a74a37981f1e, State: Running, Role: LEADER
I20260925 15:02:17.695420 29982 consensus_queue.cc:237] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } }
I20260925 15:02:17.696170 29984 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f536630d0793400692a3a74a37981f1e. Latest consensus state: current_term: 1 leader_uuid: "f536630d0793400692a3a74a37981f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } } }
I20260925 15:02:17.696352 29984 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:17.696187 29983 sys_catalog.cc:455] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f536630d0793400692a3a74a37981f1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f536630d0793400692a3a74a37981f1e" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 44027 } } }
I20260925 15:02:17.696455 29983 sys_catalog.cc:458] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:17.696554 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:17.690334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49001 (local address 127.8.13.254:44027)
0925 15:02:17.690541 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:17.690553 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:17.690591 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:02:17.691073 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:17.691083 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:17.691156 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:17.691317 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:17.691551 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:17.693013 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:17.694001 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:17.694627 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:17.694750 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:17.695689 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:17.695693 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:17.695694 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:17.695936 (+ 242us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:17.695980 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:17.695997 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:17.696093 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:17.696262 (+ 169us) server_negotiation.cc:299] Negotiation successful
0925 15:02:17.696295 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":100,"threads_started":1}
I20260925 15:02:17.696707 29996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:17.697546 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:17.697652 29996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:17.699128 29996 catalog_manager.cc:1347] Generated new cluster ID: f0655df56ba443cc8ddfdeae5fdc189b
I20260925 15:02:17.699183 29996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:17.703208 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:17.699193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43763 (local address 127.8.13.254:44027)
0925 15:02:17.699239 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:17.699241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:17.699694 (+ 453us) server_negotiation.cc:407] Connection header received
0925 15:02:17.699796 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:17.699798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:17.699810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:17.699878 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:17.700038 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:17.700779 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:17.701796 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:17.702119 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:17.702191 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:17.702936 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:17.702938 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:17.702938 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:17.702944 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:17.702970 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:17.702976 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:17.703040 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:17.703107 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:02:17.703119 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260925 15:02:17.710333 29996 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:17.710870 29996 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:17.718531 29996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f536630d0793400692a3a74a37981f1e: Generated new TSK 0
I20260925 15:02:17.718727 29996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:17.727883 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 29909
W20260925 15:02:17.731618 29894 connection.cc:535] client connection to 127.8.13.254:44027 recv error: Network error: failed to read from TLS socket (remote: 127.8.13.254:44027): Connection reset by peer (error 104)
2026-09-25T15:02:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:17 dist-test-slave-15hz krb5kdc[30008](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:17 dist-test-slave-15hz krb5kdc[30008](info): set up 2 sockets
Sep 25 15:02:17 dist-test-slave-15hz krb5kdc[30008](info): commencing operation
krb5kdc: starting...
W20260925 15:02:18.671394 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:18 dist-test-slave-15hz krb5kdc[30008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:18.785880 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37663
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:33417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37663
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:18.867360 30021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:18.867631 30021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:18.867669 30021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:18.868870 30021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:18.868919 30021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:18.868933 30021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:18.868944 30021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:18.868955 30021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:18.870528 30021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:33417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37663
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:18.870795 30021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:18.871021 30021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:18.875790 30026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:18.875784 30029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:18.875808 30021 server_base.cc:1034] running on GCE node
W20260925 15:02:18.875808 30027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:18.876174 30021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:18.876451 30021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:18.877633 30021 hybrid_clock.cc:648] HybridClock initialized: now 1790348538877613 us; error 31 us; skew 500 ppm
Sep 25 15:02:18 dist-test-slave-15hz krb5kdc[30008](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348538, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:18.880167 30021 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:18.880597 30021 webserver.cc:460] Webserver started at http://127.8.13.254:36059/ using document root <none> and password file <none>
I20260925 15:02:18.880795 30021 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:18.880846 30021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:18.880978 30021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:18.882050 30021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "58789fcf9ede46cda12c98c1239b9c0c"
format_stamp: "Formatted at 2026-09-25 15:02:18 on dist-test-slave-15hz"
server_key: "9812aea58afddad95b48606d48d9c97a"
server_key_iv: "5a6b5761ce75fc2abdebb98613a0821d"
server_key_version: "encryptionkey@0"
I20260925 15:02:18.882388 30021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "58789fcf9ede46cda12c98c1239b9c0c"
format_stamp: "Formatted at 2026-09-25 15:02:18 on dist-test-slave-15hz"
server_key: "9812aea58afddad95b48606d48d9c97a"
server_key_iv: "5a6b5761ce75fc2abdebb98613a0821d"
server_key_version: "encryptionkey@0"
I20260925 15:02:18.883721 30021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260925 15:02:18.884486 30036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:18.884649 30021 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:18.884732 30021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "58789fcf9ede46cda12c98c1239b9c0c"
format_stamp: "Formatted at 2026-09-25 15:02:18 on dist-test-slave-15hz"
server_key: "9812aea58afddad95b48606d48d9c97a"
server_key_iv: "5a6b5761ce75fc2abdebb98613a0821d"
server_key_version: "encryptionkey@0"
I20260925 15:02:18.884794 30021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:18.888123 30021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:18.889933 30021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:18.890075 30021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:18.893857 30021 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:37663
I20260925 15:02:18.893926 30088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:37663 every 8 connection(s)
I20260925 15:02:18.894236 30021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:18.895195 30089 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:18.897562 30089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: Bootstrap starting.
I20260925 15:02:18.898140 30089 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:18.898388 30089 log.cc:824] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:18.899067 30089 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: No bootstrap required, opened a new log
I20260925 15:02:18.900732 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30021
I20260925 15:02:18.900851 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:18.900811 30089 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } }
I20260925 15:02:18.900986 30089 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:18.901073 8247 external_mini_cluster.cc:1428] Setting key b238848fa0d7f0f371624a4762f3e350
I20260925 15:02:18.901072 30089 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58789fcf9ede46cda12c98c1239b9c0c, State: Initialized, Role: FOLLOWER
I20260925 15:02:18.901237 30089 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } }
I20260925 15:02:18.901309 30089 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:18.901342 30089 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:18.901391 30089 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:18.902278 30089 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } }
I20260925 15:02:18.902388 30089 leader_election.cc:302] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58789fcf9ede46cda12c98c1239b9c0c; no voters:
I20260925 15:02:18.902545 30089 leader_election.cc:288] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:18.902581 30094 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:18.902758 30094 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [term 1 LEADER]: Becoming Leader. State: Replica: 58789fcf9ede46cda12c98c1239b9c0c, State: Running, Role: LEADER
I20260925 15:02:18.902839 30089 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:18.902853 30094 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } }
I20260925 15:02:18.903247 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:18.901555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51199 (local address 127.8.13.254:37663)
0925 15:02:18.901729 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:18.901739 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:18.901775 (+ 36us) server_negotiation.cc:407] Connection header received
0925 15:02:18.901892 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:18.901898 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:18.901921 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:18.902042 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:18.902770 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:18.902775 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:18.902775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:18.902781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:18.902828 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:18.902843 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:18.902887 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:18.903029 (+ 142us) server_negotiation.cc:299] Negotiation successful
0925 15:02:18.903076 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260925 15:02:18.903450 30096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58789fcf9ede46cda12c98c1239b9c0c. Latest consensus state: current_term: 1 leader_uuid: "58789fcf9ede46cda12c98c1239b9c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } } }
I20260925 15:02:18.903568 30096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:18.903435 30095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58789fcf9ede46cda12c98c1239b9c0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58789fcf9ede46cda12c98c1239b9c0c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37663 } } }
I20260925 15:02:18.903786 30095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:18.904564 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 15:02:18.905912 30110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:18.906013 30110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:18.906070 30103 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:18.906207 30103 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:18.907920 30103 catalog_manager.cc:1347] Generated new cluster ID: 64774ac4803144259eef270a2ade2f10
I20260925 15:02:18.907976 30103 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:18.908070 30093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:18.905971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56925 (local address 127.8.13.254:37663)
0925 15:02:18.906014 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:18.906016 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:18.907018 (+ 1002us) server_negotiation.cc:407] Connection header received
0925 15:02:18.907081 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:18.907082 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:18.907088 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:18.907156 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:18.907859 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:18.907860 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:18.907860 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:18.907865 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:18.907882 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:18.907886 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:18.907934 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:18.907988 (+ 54us) server_negotiation.cc:299] Negotiation successful
0925 15:02:18.908002 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:02:18.932021 30103 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:18.932205 30103 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:18.935154 30103 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58789fcf9ede46cda12c98c1239b9c0c: Generated new TSK 0
I20260925 15:02:18.935344 30103 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:18.944487 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30021
2026-09-25T15:02:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:18 dist-test-slave-15hz krb5kdc[30120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:18 dist-test-slave-15hz krb5kdc[30120](info): set up 2 sockets
Sep 25 15:02:18 dist-test-slave-15hz krb5kdc[30120](info): commencing operation
krb5kdc: starting...
W20260925 15:02:19.895381 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:19 dist-test-slave-15hz krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:20.008664 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46863
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:41305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:46863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:20.090322 30133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:20.090579 30133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:20.090615 30133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:20.091892 30133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:20.091948 30133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:20.091967 30133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:20.091985 30133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:20.092000 30133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:20.093897 30133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:41305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:46863
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:46863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30133
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:20.094215 30133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:20.094457 30133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:20.099290 30138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:20.099290 30141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:20.099326 30133 server_base.cc:1034] running on GCE node
W20260925 15:02:20.099337 30139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:20.099764 30133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:20.100059 30133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:20.101222 30133 hybrid_clock.cc:648] HybridClock initialized: now 1790348540101208 us; error 31 us; skew 500 ppm
Sep 25 15:02:20 dist-test-slave-15hz krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348540, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:20.109776 30133 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:20.110268 30133 webserver.cc:460] Webserver started at http://127.8.13.254:42075/ using document root <none> and password file <none>
I20260925 15:02:20.110500 30133 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:20.110553 30133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:20.110698 30133 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:20.111855 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879"
format_stamp: "Formatted at 2026-09-25 15:02:20 on dist-test-slave-15hz"
server_key: "c6deb9d94cbf29c290d856dccd3e23cf"
server_key_iv: "2498588000def97654dd3c1c02458b6a"
server_key_version: "encryptionkey@0"
I20260925 15:02:20.112211 30133 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879"
format_stamp: "Formatted at 2026-09-25 15:02:20 on dist-test-slave-15hz"
server_key: "c6deb9d94cbf29c290d856dccd3e23cf"
server_key_iv: "2498588000def97654dd3c1c02458b6a"
server_key_version: "encryptionkey@0"
I20260925 15:02:20.113619 30133 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:20.114409 30148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:20.114596 30133 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:20.114686 30133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879"
format_stamp: "Formatted at 2026-09-25 15:02:20 on dist-test-slave-15hz"
server_key: "c6deb9d94cbf29c290d856dccd3e23cf"
server_key_iv: "2498588000def97654dd3c1c02458b6a"
server_key_version: "encryptionkey@0"
I20260925 15:02:20.114750 30133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:20.158357 30133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:20.160576 30133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:20.160738 30133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:20.165089 30133 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:46863
I20260925 15:02:20.165144 30200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:46863 every 8 connection(s)
I20260925 15:02:20.165548 30133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:20.166566 30201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:20.169052 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879: Bootstrap starting.
I20260925 15:02:20.169661 30201 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:20.169934 30201 log.cc:824] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:20.170652 30201 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879: No bootstrap required, opened a new log
I20260925 15:02:20.172343 30201 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } }
I20260925 15:02:20.172509 30201 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:20.172535 30201 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c2a9a4bc9c4b2d9e8989dccd8e5879, State: Initialized, Role: FOLLOWER
I20260925 15:02:20.172690 30201 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } }
I20260925 15:02:20.172772 30201 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:20.172811 30201 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:20.172865 30201 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:20.173753 30201 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } }
I20260925 15:02:20.173853 30201 leader_election.cc:302] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01c2a9a4bc9c4b2d9e8989dccd8e5879; no voters:
I20260925 15:02:20.174031 30201 leader_election.cc:288] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:20.174073 30204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:20.174219 30204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [term 1 LEADER]: Becoming Leader. State: Replica: 01c2a9a4bc9c4b2d9e8989dccd8e5879, State: Running, Role: LEADER
I20260925 15:02:20.174340 30201 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:20.174355 30204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } }
I20260925 15:02:20.174424 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30133
I20260925 15:02:20.174540 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:20.174746 8247 external_mini_cluster.cc:1428] Setting key ecf493f3669503e8baf27cf6e71409e5
I20260925 15:02:20.174823 30206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01c2a9a4bc9c4b2d9e8989dccd8e5879. Latest consensus state: current_term: 1 leader_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } } }
I20260925 15:02:20.174899 30206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:20.175851 30211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:20.175956 30205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01c2a9a4bc9c4b2d9e8989dccd8e5879" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 46863 } } }
I20260925 15:02:20.176070 30205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:20.176489 30211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:20.178158 30211 catalog_manager.cc:1347] Generated new cluster ID: 71b1370ff48f4eff8a05f1e2fdb7b267
I20260925 15:02:20.178205 30211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:02:20 dist-test-slave-15hz krb5kdc[30120](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:20.186738 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.175448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41053 (local address 127.8.13.254:46863)
0925 15:02:20.175682 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:20.175692 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:20.175730 (+ 38us) server_negotiation.cc:407] Connection header received
0925 15:02:20.175867 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:20.175875 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:20.175951 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:20.176106 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:20.177040 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.178248 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.179301 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.179932 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.180027 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:20.182832 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:20.182838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:20.182838 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:20.182852 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:20.184984 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.185363 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.185367 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.185367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.185519 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.185713 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.185714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.185714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.186093 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:20.186266 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:20.186342 (+ 76us) server_negotiation.cc:299] Negotiation successful
0925 15:02:20.186504 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260925 15:02:20.187922 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:20.195647 30211 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:20.196349 30211 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:20.199270 30216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:20.191917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47427 (local address 127.8.13.254:46863)
0925 15:02:20.191957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:20.191959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:20.192164 (+ 205us) server_negotiation.cc:407] Connection header received
0925 15:02:20.192205 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:20.192206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:20.192219 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:20.192287 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:20.192880 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.193659 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.194653 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:20.194961 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:20.195038 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:20.196507 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:20.196509 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:20.196509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:20.196517 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:20.198145 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.198514 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.198515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.198515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.198578 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:20.198743 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:20.198743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:20.198744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:20.198876 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:20.198994 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:20.199073 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:02:20.199172 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:02:20.200083 30211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01c2a9a4bc9c4b2d9e8989dccd8e5879: Generated new TSK 0
I20260925 15:02:20.200258 30211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:20.205013 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30133
2026-09-25T15:02:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1260 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:20 dist-test-slave-15hz krb5kdc[30232](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:20 dist-test-slave-15hz krb5kdc[30232](info): set up 2 sockets
Sep 25 15:02:20 dist-test-slave-15hz krb5kdc[30232](info): commencing operation
krb5kdc: starting...
W20260925 15:02:21.151407 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:21 dist-test-slave-15hz krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:21.268164 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34873
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:38009
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:34873
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:21.350212 30245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:21.350420 30245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:21.350441 30245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:21.351649 30245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:21.351696 30245 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:21.351708 30245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:21.351719 30245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:21.351730 30245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:21.353288 30245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:38009
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:34873
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:34873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30245
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:21.353542 30245 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:21.353765 30245 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:21.358532 30251 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:21.358644 30245 server_base.cc:1034] running on GCE node
W20260925 15:02:21.358560 30253 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:21.358592 30250 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:21.359102 30245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:21.359419 30245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:21.360589 30245 hybrid_clock.cc:648] HybridClock initialized: now 1790348541360575 us; error 30 us; skew 500 ppm
Sep 25 15:02:21 dist-test-slave-15hz krb5kdc[30232](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348541, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:21.363164 30245 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:21.363662 30245 webserver.cc:460] Webserver started at http://127.8.13.254:36637/ using document root <none> and password file <none>
I20260925 15:02:21.363864 30245 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:21.363904 30245 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:21.364001 30245 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:21.365051 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "abb2090f2daf4c92b2c8ea2b1083963c"
format_stamp: "Formatted at 2026-09-25 15:02:21 on dist-test-slave-15hz"
server_key: "4febc447fd1039e52a9d3308f87f3b90"
server_key_iv: "31951252a2286c92a8c6f116b2ee55bf"
server_key_version: "encryptionkey@0"
I20260925 15:02:21.365351 30245 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "abb2090f2daf4c92b2c8ea2b1083963c"
format_stamp: "Formatted at 2026-09-25 15:02:21 on dist-test-slave-15hz"
server_key: "4febc447fd1039e52a9d3308f87f3b90"
server_key_iv: "31951252a2286c92a8c6f116b2ee55bf"
server_key_version: "encryptionkey@0"
I20260925 15:02:21.366623 30245 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:02:21.367498 30260 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:21.367722 30245 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260925 15:02:21.367812 30245 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "abb2090f2daf4c92b2c8ea2b1083963c"
format_stamp: "Formatted at 2026-09-25 15:02:21 on dist-test-slave-15hz"
server_key: "4febc447fd1039e52a9d3308f87f3b90"
server_key_iv: "31951252a2286c92a8c6f116b2ee55bf"
server_key_version: "encryptionkey@0"
I20260925 15:02:21.367871 30245 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:21.386286 30245 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:21.388562 30245 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:21.388679 30245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:21.392426 30245 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:34873
I20260925 15:02:21.392472 30312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:34873 every 8 connection(s)
I20260925 15:02:21.392799 30245 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:21.393020 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30245
I20260925 15:02:21.393134 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:21.393357 8247 external_mini_cluster.cc:1428] Setting key 65c1ee6dd73a13cf00b71922d25511ba
I20260925 15:02:21.393802 30313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:21.397039 30313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c: Bootstrap starting.
I20260925 15:02:21.397702 30313 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:21.398030 30313 log.cc:824] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:21.398713 30313 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c: No bootstrap required, opened a new log
I20260925 15:02:21.399303 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:21.393872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46227 (local address 127.8.13.254:34873)
0925 15:02:21.394067 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:21.394076 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:21.394106 (+ 30us) server_negotiation.cc:407] Connection header received
0925 15:02:21.394601 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:21.394611 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:21.394679 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:21.394829 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:21.395035 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:21.396415 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:21.397276 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:21.397891 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:21.398003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:21.398643 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:21.398648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:21.398649 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:21.398851 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:21.398890 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:21.398905 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:21.398969 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:21.399025 (+ 56us) server_negotiation.cc:299] Negotiation successful
0925 15:02:21.399094 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":87,"threads_started":1}
I20260925 15:02:21.400454 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:21.400812 30313 raft_consensus.cc:348] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } }
I20260925 15:02:21.401010 30313 raft_consensus.cc:374] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:21.401044 30313 raft_consensus.cc:729] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abb2090f2daf4c92b2c8ea2b1083963c, State: Initialized, Role: FOLLOWER
I20260925 15:02:21.401160 30313 consensus_queue.cc:260] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } }
I20260925 15:02:21.401240 30313 raft_consensus.cc:388] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:21.401275 30313 raft_consensus.cc:482] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:21.401319 30313 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:21.402066 30313 raft_consensus.cc:504] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } }
I20260925 15:02:21.402185 30313 leader_election.cc:302] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abb2090f2daf4c92b2c8ea2b1083963c; no voters:
I20260925 15:02:21.402364 30313 leader_election.cc:288] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:21.402392 30323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:21.402508 30323 raft_consensus.cc:686] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [term 1 LEADER]: Becoming Leader. State: Replica: abb2090f2daf4c92b2c8ea2b1083963c, State: Running, Role: LEADER
I20260925 15:02:21.402590 30323 consensus_queue.cc:237] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } }
I20260925 15:02:21.402622 30313 sys_catalog.cc:564] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:21.402899 30323 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } } }
I20260925 15:02:21.403054 30323 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:21.403219 30323 sys_catalog.cc:455] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [sys.catalog]: SysCatalogTable state changed. Reason: New leader abb2090f2daf4c92b2c8ea2b1083963c. Latest consensus state: current_term: 1 leader_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abb2090f2daf4c92b2c8ea2b1083963c" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 34873 } } }
I20260925 15:02:21.403321 30323 sys_catalog.cc:458] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:21.403725 30330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:21.404243 30330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:21.405757 30316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:21.401972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49629 (local address 127.8.13.254:34873)
0925 15:02:21.402021 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:21.402023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:21.402144 (+ 121us) server_negotiation.cc:407] Connection header received
0925 15:02:21.402237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:21.402238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:21.402249 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:21.402303 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:21.402497 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:21.403156 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:21.404208 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:21.404472 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:21.404530 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:21.405524 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:21.405525 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:21.405526 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:21.405530 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:21.405544 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:21.405547 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:21.405590 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:21.405657 (+ 67us) server_negotiation.cc:299] Negotiation successful
0925 15:02:21.405684 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:02:21.405898 30330 catalog_manager.cc:1347] Generated new cluster ID: cde11ffcff2d4ce584f2e55c1355572c
I20260925 15:02:21.405937 30330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:21.425634 30330 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:21.426208 30330 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:21.430444 30330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abb2090f2daf4c92b2c8ea2b1083963c: Generated new TSK 0
I20260925 15:02:21.430634 30330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:21.438431 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30245
2026-09-25T15:02:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:21 dist-test-slave-15hz krb5kdc[30344](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:21 dist-test-slave-15hz krb5kdc[30344](info): set up 2 sockets
Sep 25 15:02:21 dist-test-slave-15hz krb5kdc[30344](info): commencing operation
krb5kdc: starting...
W20260925 15:02:22.395803 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.940s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:22 dist-test-slave-15hz krb5kdc[30344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:22.511765 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43169
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:39775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:43169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:22.593498 30357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:22.593732 30357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:22.593766 30357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:22.594944 30357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:22.594988 30357 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:22.595005 30357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:22.595021 30357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:22.595036 30357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:22.596573 30357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:39775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:43169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:43169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30357
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:22.596830 30357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:22.597043 30357 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:22.601877 30365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:22.601883 30363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:22.601912 30362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:22.602223 30357 server_base.cc:1034] running on GCE node
I20260925 15:02:22.602461 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:22.602744 30357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:22.603909 30357 hybrid_clock.cc:648] HybridClock initialized: now 1790348542603885 us; error 34 us; skew 500 ppm
Sep 25 15:02:22 dist-test-slave-15hz krb5kdc[30344](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348542, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:22.606432 30357 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:22.606848 30357 webserver.cc:460] Webserver started at http://127.8.13.254:34355/ using document root <none> and password file <none>
I20260925 15:02:22.607054 30357 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:22.607098 30357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:22.607206 30357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:22.608307 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "1d2faefc4c7d44c7a716d26453f47b6d"
format_stamp: "Formatted at 2026-09-25 15:02:22 on dist-test-slave-15hz"
server_key: "07586c3eaf7812d82d96f0dc870d60a3"
server_key_iv: "8b1ab58d8ed1fbcc062f7066531d0d8f"
server_key_version: "encryptionkey@0"
I20260925 15:02:22.608620 30357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "1d2faefc4c7d44c7a716d26453f47b6d"
format_stamp: "Formatted at 2026-09-25 15:02:22 on dist-test-slave-15hz"
server_key: "07586c3eaf7812d82d96f0dc870d60a3"
server_key_iv: "8b1ab58d8ed1fbcc062f7066531d0d8f"
server_key_version: "encryptionkey@0"
I20260925 15:02:22.609833 30357 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:22.610618 30372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:22.610900 30357 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:22.611014 30357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "1d2faefc4c7d44c7a716d26453f47b6d"
format_stamp: "Formatted at 2026-09-25 15:02:22 on dist-test-slave-15hz"
server_key: "07586c3eaf7812d82d96f0dc870d60a3"
server_key_iv: "8b1ab58d8ed1fbcc062f7066531d0d8f"
server_key_version: "encryptionkey@0"
I20260925 15:02:22.611086 30357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:22.614513 30357 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:22.616729 30357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:22.616869 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:22.620886 30357 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:43169
I20260925 15:02:22.620940 30424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:43169 every 8 connection(s)
I20260925 15:02:22.621304 30357 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:22.622412 30425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:22.625056 30425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d: Bootstrap starting.
I20260925 15:02:22.625672 30425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:22.625926 30425 log.cc:824] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:22.626446 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30357
I20260925 15:02:22.626554 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:22.626713 8247 external_mini_cluster.cc:1428] Setting key 2d724614855238f207bcdaf6ad274a89
I20260925 15:02:22.626752 30425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d: No bootstrap required, opened a new log
I20260925 15:02:22.628989 30425 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } }
I20260925 15:02:22.629182 30425 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260925 15:02:22.629220 30425 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2faefc4c7d44c7a716d26453f47b6d, State: Initialized, Role: FOLLOWER
I20260925 15:02:22.629382 30425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } }
I20260925 15:02:22.629490 30425 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260925 15:02:22.629523 30425 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260925 15:02:22.629570 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:22.627285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37933 (local address 127.8.13.254:43169)
0925 15:02:22.627506 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:22.627516 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:22.627575 (+ 59us) server_negotiation.cc:407] Connection header received
0925 15:02:22.628143 (+ 568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:22.628154 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:22.628193 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:22.628353 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:22.628957 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:22.628961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:22.628961 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:22.628965 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:22.629021 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:22.629036 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:22.629107 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:22.629170 (+ 63us) server_negotiation.cc:299] Negotiation successful
0925 15:02:22.629213 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":105,"threads_started":1}
I20260925 15:02:22.629573 30425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:22.630343 30425 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } }
I20260925 15:02:22.630453 30425 leader_election.cc:302] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1d2faefc4c7d44c7a716d26453f47b6d; no voters:
I20260925 15:02:22.630487 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:22.630636 30425 leader_election.cc:288] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260925 15:02:22.630664 30430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:22.630807 30430 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [term 1 LEADER]: Becoming Leader. State: Replica: 1d2faefc4c7d44c7a716d26453f47b6d, State: Running, Role: LEADER
I20260925 15:02:22.630861 30425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:22.630934 30430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } }
I20260925 15:02:22.631268 30430 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1d2faefc4c7d44c7a716d26453f47b6d. Latest consensus state: current_term: 1 leader_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } } }
I20260925 15:02:22.631321 30432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1d2faefc4c7d44c7a716d26453f47b6d" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 43169 } } }
I20260925 15:02:22.631402 30432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:22.631621 30430 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:22.631778 30438 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:22.632290 30438 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:22.633234 30429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:22.632018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54519 (local address 127.8.13.254:43169)
0925 15:02:22.632053 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:22.632055 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:22.632220 (+ 165us) server_negotiation.cc:407] Connection header received
0925 15:02:22.632312 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:22.632313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:22.632317 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:22.632346 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:22.633010 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:22.633011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:22.633011 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0925 15:02:22.633016 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:22.633031 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0925 15:02:22.633034 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:22.633065 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:22.633164 (+ 99us) server_negotiation.cc:299] Negotiation successful
0925 15:02:22.633175 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:22.633874 30438 catalog_manager.cc:1347] Generated new cluster ID: ae2576b476f7444091cca18ab7c46aa8
I20260925 15:02:22.633927 30438 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:22.652755 30438 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:22.652969 30438 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:22.666934 30438 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1d2faefc4c7d44c7a716d26453f47b6d: Generated new TSK 0
I20260925 15:02:22.667191 30438 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:22.675653 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30357
2026-09-25T15:02:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1237 ms)
[----------] 18 tests from AuthTokenIssuingTest (22069 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:22 dist-test-slave-15hz krb5kdc[30456](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:22 dist-test-slave-15hz krb5kdc[30456](info): set up 2 sockets
Sep 25 15:02:22 dist-test-slave-15hz krb5kdc[30456](info): commencing operation
krb5kdc: starting...
W20260925 15:02:23.631397 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:23 dist-test-slave-15hz krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:23.742358 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42267
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:23.823553 30469 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:23.823822 30469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:23.823839 30469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:23.823860 30469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:23.825037 30469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:23.825075 30469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:23.825089 30469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:23.825109 30469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:23.825119 30469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:23.826613 30469 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:23.826835 30469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:23.827018 30469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:23.832842 30474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:23.832844 30475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:23.832849 30477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:23.833848 30469 server_base.cc:1034] running on GCE node
I20260925 15:02:23.834193 30469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:23.834555 30469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:23.835697 30469 hybrid_clock.cc:648] HybridClock initialized: now 1790348543835650 us; error 75 us; skew 500 ppm
Sep 25 15:02:23 dist-test-slave-15hz krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:23.838523 30469 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:23.839010 30469 webserver.cc:460] Webserver started at http://127.8.13.254:33033/ using document root <none> and password file <none>
I20260925 15:02:23.839221 30469 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:23.839270 30469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:23.839398 30469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:23.840516 30469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "895a3b7f860447f695463c8cc7e14ad2"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "03c4581860e444bb8ffea3dfbc315c41"
server_key_iv: "696e7d0251fe09d2a908348dae2f47cd"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.840873 30469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "895a3b7f860447f695463c8cc7e14ad2"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "03c4581860e444bb8ffea3dfbc315c41"
server_key_iv: "696e7d0251fe09d2a908348dae2f47cd"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.842151 30469 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:23.842911 30484 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:23.843112 30469 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:23.843206 30469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "895a3b7f860447f695463c8cc7e14ad2"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "03c4581860e444bb8ffea3dfbc315c41"
server_key_iv: "696e7d0251fe09d2a908348dae2f47cd"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.843288 30469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:23.852366 30469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:23.854622 30469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:23.854772 30469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:23.858784 30469 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:42267
I20260925 15:02:23.858829 30536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:42267 every 8 connection(s)
I20260925 15:02:23.859193 30469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:23.860195 30537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:23.861747 30537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:23.867250 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30469
I20260925 15:02:23.867340 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:23.867520 8247 external_mini_cluster.cc:1428] Setting key 29ee72324ace6e91a5d489f5961b766b
Sep 25 15:02:23 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:23.872017 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.862449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42267 (local address 127.0.0.1:55818)
0925 15:02:23.862652 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.862695 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:23.863217 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:23.863657 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:23.863661 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:23.863981 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:23.864184 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:23.864190 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:23.865251 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:23.865252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:23.865970 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:23.865972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:23.866503 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:23.866504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:23.866560 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:23.867098 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:23.867107 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:23.868751 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:23.870546 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:23.870552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:23.870555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:23.870786 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:23.871009 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:23.871009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:23.871010 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:23.871095 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:23.871419 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:23.871423 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:23.871494 (+ 71us) client_negotiation.cc:769] Sending connection context
0925 15:02:23.871608 (+ 114us) client_negotiation.cc:240] Negotiation successful
0925 15:02:23.871832 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":72,"thread_start_us":91,"threads_started":1}
I20260925 15:02:23.872020 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.862494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55818 (local address 127.8.13.254:42267)
0925 15:02:23.862704 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:23.862708 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:23.862789 (+ 81us) server_negotiation.cc:407] Connection header received
0925 15:02:23.863393 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:23.863395 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:23.863424 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:23.863508 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:23.864291 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:23.865156 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:23.866078 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:23.866416 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:23.866504 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:23.868846 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:23.868853 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:23.868854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:23.868861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:23.870467 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:23.870865 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:23.870867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:23.870868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:23.870947 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:23.871146 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:23.871147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:23.871147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:23.871361 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:23.871444 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:23.871640 (+ 196us) server_negotiation.cc:299] Negotiation successful
0925 15:02:23.871755 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":65,"threads_started":1}
I20260925 15:02:23.873031 30537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:23.873425 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.873309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35664)
0925 15:02:23.873352 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.873394 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:02:23.873579 30486 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
I20260925 15:02:23.874006 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.873915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35666)
0925 15:02:23.873956 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.873982 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:02:23.874100 30537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39061: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.13.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
I20260925 15:02:23.890509 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:39061
--webserver_interface=127.8.13.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:23.923345 30537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } attempt: 1
I20260925 15:02:23.923942 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.923753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35674)
0925 15:02:23.923835 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.923873 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:02:23.924310 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.924214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35680)
0925 15:02:23.924259 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.924283 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:02:23.924424 30537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39061: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
W20260925 15:02:23.973500 30544 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:23.973718 30544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:23.973734 30544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:23.973755 30544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:23.974924 30544 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:23.974969 30544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:23.974982 30544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:23.974993 30544 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:23.975003 30544 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:23.976495 30544 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:39061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:23.976697 30544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:23.976873 30544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:23.981451 30552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:23.981447 30550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:23.981482 30549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:23.981715 30544 server_base.cc:1034] running on GCE node
I20260925 15:02:23.981956 30544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:23.982239 30544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:23.983379 30544 hybrid_clock.cc:648] HybridClock initialized: now 1790348543983351 us; error 44 us; skew 500 ppm
Sep 25 15:02:23 dist-test-slave-15hz krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:23.985911 30544 init.cc:377] Logged in from keytab as kudu/127.8.13.253@KRBTEST.COM (short username kudu)
I20260925 15:02:23.986359 30544 webserver.cc:460] Webserver started at http://127.8.13.253:46365/ using document root <none> and password file <none>
I20260925 15:02:23.986559 30544 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:23.986606 30544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:23.986732 30544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:23.987852 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data/instance:
uuid: "bc4cdce86741462597ff714c1f1faca8"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "68d70205a6512631b423076a6178f7bd"
server_key_iv: "3ec8db3415e487620283deea72ee9147"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.988184 30544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal/instance:
uuid: "bc4cdce86741462597ff714c1f1faca8"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "68d70205a6512631b423076a6178f7bd"
server_key_iv: "3ec8db3415e487620283deea72ee9147"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.989457 30544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:23.990263 30559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:23.990430 30544 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:23.990510 30544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "bc4cdce86741462597ff714c1f1faca8"
format_stamp: "Formatted at 2026-09-25 15:02:23 on dist-test-slave-15hz"
server_key: "68d70205a6512631b423076a6178f7bd"
server_key_iv: "3ec8db3415e487620283deea72ee9147"
server_key_version: "encryptionkey@0"
I20260925 15:02:23.990569 30544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:23.992605 30537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } attempt: 2
I20260925 15:02:23.993008 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.992885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35694)
0925 15:02:23.992942 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.992977 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260925 15:02:23.993373 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:23.993301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35706)
0925 15:02:23.993335 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:23.993358 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:02:23.993448 30537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39061: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39061: connect: Connection refused (error 111)
I20260925 15:02:24.015756 30544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:24.017982 30544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:24.018139 30544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:24.021847 30544 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.253:39061
I20260925 15:02:24.021893 30611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.253:39061 every 8 connection(s)
I20260925 15:02:24.022226 30544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
I20260925 15:02:24.023195 30612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:24.024766 30612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:24.025583 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30544
I20260925 15:02:24.025667 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-1/wal/instance
I20260925 15:02:24.025832 8247 external_mini_cluster.cc:1428] Setting key 42fd282f8c7b0c1b9e092d404b52dd97
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.252@KRBTEST.COM; defaulting to no policy
I20260925 15:02:24.033862 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.025465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55832 (local address 127.8.13.254:42267)
0925 15:02:24.025527 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.025530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.025729 (+ 199us) server_negotiation.cc:407] Connection header received
0925 15:02:24.026256 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.026258 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.026271 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.026316 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.026944 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.027430 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.028423 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.028625 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.028680 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.031080 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.031082 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.031083 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.031090 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.032458 (+ 1368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.032934 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.032934 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.032935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.032984 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.033222 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.033223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.033223 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.033344 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.033402 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.033680 (+ 278us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.033766 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260925 15:02:24.034039 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.025421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42267 (local address 127.0.0.1:55832)
0925 15:02:24.025589 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.025646 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.026159 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.026397 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.026402 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.026652 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.026854 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.026859 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.027513 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.027514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.028339 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.028342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.028714 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.028716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.028829 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.029302 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.029310 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.030951 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.032549 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.032558 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.032560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.032857 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.033036 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.033036 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.033037 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.033168 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.033404 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.033412 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.033557 (+ 145us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.033653 (+ 96us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.033812 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260925 15:02:24.034550 30612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.252@KRBTEST.COM" created.
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:02:24.041792 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.034834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35712)
0925 15:02:24.034879 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.034890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.034932 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.035254 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.035255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.035364 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.035415 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.035416 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.036430 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.036431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.036938 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.036940 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.037468 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.037469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.037540 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.037941 (+ 401us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.037947 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.039095 (+ 1148us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.040812 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.040813 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.040814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.040976 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.041150 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.041151 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.041151 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.041202 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.041454 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.041455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.041495 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.041554 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.041661 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:02:24.041810 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.034979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35712 (local address 127.8.13.253:39061)
0925 15:02:24.035127 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.035131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.035148 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:02:24.035166 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.035168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.035179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.035238 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.035463 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.036346 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.037013 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.037402 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.037459 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.039195 (+ 1736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.039205 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.039205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.039212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.040738 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.041043 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.041046 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.041047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.041100 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.041249 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.041250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.041250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.041397 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.041477 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.041582 (+ 105us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.041712 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260925 15:02:24.042387 30612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:24.042732 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.042616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50294)
0925 15:02:24.042656 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.042705 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:02:24.042860 30561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
I20260925 15:02:24.043294 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.043207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50306)
0925 15:02:24.043247 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.043271 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 15:02:24.043409 30612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:38455: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
I20260925 15:02:24.050705 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:38455
--webserver_interface=127.8.13.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:43491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:24.084606 30537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } attempt: 3
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:02:24.091378 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.085005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35726)
0925 15:02:24.085057 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.085069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.085113 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.085288 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.085289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.085489 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.085567 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.085569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.086247 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.086249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.086790 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.086791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.087159 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.087160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.087234 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.087731 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.087736 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.088931 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.090418 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.090419 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.090420 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.090592 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.090769 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.090770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.090770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.090829 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.091048 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.091049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.091101 (+ 52us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.091161 (+ 60us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.091272 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260925 15:02:24.091378 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.085102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35726 (local address 127.8.13.253:39061)
0925 15:02:24.085171 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.085173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.085182 (+ 9us) server_negotiation.cc:407] Connection header received
0925 15:02:24.085210 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.085210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.085223 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.085275 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.085653 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.086168 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.086868 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.087096 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.087160 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.089035 (+ 1875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.089037 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.089037 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.089046 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.090330 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.090658 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.090659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.090660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.090716 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.090875 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.090876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.090876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.090994 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.091048 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.091189 (+ 141us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.091272 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260925 15:02:24.091801 30537 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:24.092180 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.092076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50316)
0925 15:02:24.092124 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.092156 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260925 15:02:24.092511 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.092423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50328)
0925 15:02:24.092464 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.092487 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:02:24.092595 30537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:38455: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
I20260925 15:02:24.092612 30612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } attempt: 1
I20260925 15:02:24.092998 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.092876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50330)
0925 15:02:24.092926 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.092962 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260925 15:02:24.093371 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.093284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50344)
0925 15:02:24.093321 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.093348 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260925 15:02:24.093472 30612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:38455: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
I20260925 15:02:24.123762 30537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } attempt: 1
I20260925 15:02:24.124248 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.124110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50354)
0925 15:02:24.124182 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.124215 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260925 15:02:24.124612 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.124536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50356)
0925 15:02:24.124566 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.124591 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260925 15:02:24.124693 30537 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:38455: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
W20260925 15:02:24.131619 30619 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:24.131853 30619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:24.131879 30619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:24.131913 30619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:24.133141 30619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:24.133190 30619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:24.133209 30619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:24.133227 30619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:24.133242 30619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:24.134783 30619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:43491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:42267,127.8.13.253:39061,127.8.13.252:38455
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:38455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:24.135042 30619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:24.135253 30619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:24.139919 30624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:24.139883 30625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:24.139883 30627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:24.140105 30619 server_base.cc:1034] running on GCE node
I20260925 15:02:24.140440 30619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:24.140736 30619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:24.141887 30619 hybrid_clock.cc:648] HybridClock initialized: now 1790348544141875 us; error 31 us; skew 500 ppm
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:24.144292 30619 init.cc:377] Logged in from keytab as kudu/127.8.13.252@KRBTEST.COM (short username kudu)
I20260925 15:02:24.144732 30619 webserver.cc:460] Webserver started at http://127.8.13.252:43121/ using document root <none> and password file <none>
I20260925 15:02:24.144930 30619 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:24.144970 30619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:24.145068 30619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:24.146165 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data/instance:
uuid: "7d3548877a3a464089bf996171223560"
format_stamp: "Formatted at 2026-09-25 15:02:24 on dist-test-slave-15hz"
server_key: "f1c9d41aece64d015016b439dea48a40"
server_key_iv: "72e326f18f08b471f90ded1b8557b46f"
server_key_version: "encryptionkey@0"
I20260925 15:02:24.146469 30619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal/instance:
uuid: "7d3548877a3a464089bf996171223560"
format_stamp: "Formatted at 2026-09-25 15:02:24 on dist-test-slave-15hz"
server_key: "f1c9d41aece64d015016b439dea48a40"
server_key_iv: "72e326f18f08b471f90ded1b8557b46f"
server_key_version: "encryptionkey@0"
I20260925 15:02:24.147786 30619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:02:24.148505 30634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:24.148670 30619 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:24.148751 30619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "7d3548877a3a464089bf996171223560"
format_stamp: "Formatted at 2026-09-25 15:02:24 on dist-test-slave-15hz"
server_key: "f1c9d41aece64d015016b439dea48a40"
server_key_iv: "72e326f18f08b471f90ded1b8557b46f"
server_key_version: "encryptionkey@0"
I20260925 15:02:24.148826 30619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:24.161648 30612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } attempt: 2
I20260925 15:02:24.162088 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.161968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50368)
0925 15:02:24.162025 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.162058 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260925 15:02:24.162416 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.162325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50370)
0925 15:02:24.162371 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.162396 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260925 15:02:24.162539 30612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:38455: Network error: Client connection negotiation failed: client connection to 127.8.13.252:38455: connect: Connection refused (error 111)
I20260925 15:02:24.177404 30619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:24.180243 30619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:24.180390 30619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:24.184077 30619 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.252:38455
I20260925 15:02:24.184106 30686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.252:38455 every 8 connection(s)
I20260925 15:02:24.184463 30619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
I20260925 15:02:24.185364 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30619
I20260925 15:02:24.185458 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790348284423390-8247-0/minicluster-data/master-2/wal/instance
I20260925 15:02:24.185511 30687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:24.185681 8247 external_mini_cluster.cc:1428] Setting key dbe3fe30c6cc672b7a3c9e13f48ea06a
I20260925 15:02:24.187605 30687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:24.195607 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.186164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.8.13.252:38455)
0925 15:02:24.186353 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.186362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.186387 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:02:24.186926 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.186934 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.186985 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.187111 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.187763 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.188851 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.189596 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.190010 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.190106 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.192381 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.192389 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.192390 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.192397 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.194120 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.194439 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.194442 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.194443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.194539 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.194725 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.194726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.194726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.194977 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.195119 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.195225 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.195350 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260925 15:02:24.195832 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.188230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:42267 (local address 127.0.0.1:55844)
0925 15:02:24.188390 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.188424 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.188492 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.188641 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.188644 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.188918 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.188984 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.188986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.189600 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.189601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.190279 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.190282 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.190680 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.190681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.190736 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.191136 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.191145 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.192768 (+ 1623us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.194790 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.194796 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.194798 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.194970 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.195148 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.195149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.195149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.195211 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.195474 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.195479 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.195524 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.195643 (+ 119us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.195746 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":87,"threads_started":1}
I20260925 15:02:24.196105 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.188277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55844 (local address 127.8.13.254:42267)
0925 15:02:24.188320 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.188322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.188487 (+ 165us) server_negotiation.cc:407] Connection header received
0925 15:02:24.188551 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.188552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.188563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.188610 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.189063 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.189521 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.190376 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.190579 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.190651 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.192846 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.192848 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.192848 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.192856 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.194717 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.195036 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.195037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.195037 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.195090 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.195266 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.195267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.195267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.195388 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.195447 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.195699 (+ 252us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.195789 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:02:24.196465 30687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:24.199820 30537 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } attempt: 2
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:24.204135 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.196742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35730 (local address 127.8.13.253:39061)
0925 15:02:24.196782 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.196783 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.196794 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:02:24.196817 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.196818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.196827 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.196869 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.197100 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.197780 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.198694 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.199000 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.199054 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.201299 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.201301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.201301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.201308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.202861 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.203233 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.203234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.203234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.203295 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.203502 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.203503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.203503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.203704 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.203771 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.203957 (+ 186us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.204037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:02:24.204433 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.196693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39061 (local address 127.0.0.1:35730)
0925 15:02:24.196735 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.196743 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.196775 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.196873 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.196874 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.197005 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.197042 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.197044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.197848 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.197849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.198616 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.198618 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.199059 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.199060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.199117 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.199506 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.199511 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.201195 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.202943 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.202944 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.202944 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.203159 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.203361 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.203361 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.203361 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.203425 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.203802 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.203803 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.203848 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.204217 (+ 369us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.204315 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 15:02:24.204885 30687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:24.204995 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.196399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55860 (local address 127.8.13.254:42267)
0925 15:02:24.196429 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.196431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.196437 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:24.196454 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.196454 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.196463 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.196504 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.197031 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.197742 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.198694 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.198982 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.199047 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.201752 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.201753 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.201754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.201761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.203304 (+ 1543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.203764 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.203765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.203765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.203819 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.203978 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.203979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.203979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.204100 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.204289 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.204836 (+ 547us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.204915 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:24.207018 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.200147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.8.13.252:38455)
0925 15:02:24.200179 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.200181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.200189 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:02:24.200246 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.200247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.200254 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.200302 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.200601 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.201151 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.202031 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.202354 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.202446 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.204356 (+ 1910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.204358 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.204358 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.204364 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.205977 (+ 1613us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.206347 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.206348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.206348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.206395 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.206565 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.206566 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.206566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.206673 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.206725 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.206858 (+ 133us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.206929 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260925 15:02:24.207022 30539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.200089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50386)
0925 15:02:24.200145 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.200155 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.200191 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.200319 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.200320 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.200464 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.200505 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.200506 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.201235 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.201236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.201915 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.201918 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.202450 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.202452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.202519 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.202844 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.202848 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.204287 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.206070 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.206072 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.206072 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.206273 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.206454 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.206455 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.206455 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.206511 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.206726 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.206727 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.206772 (+ 45us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.206831 (+ 59us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.206930 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260925 15:02:24.208933 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: Bootstrap starting.
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348544, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:24.209825 30537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:24.210145 30537 log.cc:824] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: Log is configured to *not* fsync() on all Append() calls
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:02:24.211151 30537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: No bootstrap required, opened a new log
I20260925 15:02:24.212584 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.205156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50400)
0925 15:02:24.205192 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.205202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.205407 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.205702 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.205703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.205876 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.205905 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.205907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.207126 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.207127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.207927 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.207929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.208277 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.208278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.208325 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.208680 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.208685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.209775 (+ 1090us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.211674 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.211674 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.211675 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.211834 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.211982 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.211982 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.211983 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.212036 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.212281 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.212281 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.212321 (+ 40us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.212379 (+ 58us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.212480 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260925 15:02:24.212671 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.205259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50400 (local address 127.8.13.252:38455)
0925 15:02:24.205599 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.205601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.205607 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:24.205622 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.205623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.205631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.205690 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.205964 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.207037 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.208008 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.208218 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.208267 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.209847 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.209849 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.209849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.209855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.211593 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.211888 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.211888 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.211888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.211926 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.212084 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.212084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.212085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.212225 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.212279 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.212404 (+ 125us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.212529 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":321,"thread_start_us":119,"threads_started":1}
I20260925 15:02:24.213641 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.205562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35734 (local address 127.8.13.253:39061)
0925 15:02:24.205602 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.205604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.205611 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:02:24.205632 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.205633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.205642 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.205696 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.206301 (+ 605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.207036 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.208011 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.208270 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.208336 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.210857 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.210859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.210859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.210865 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.212391 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.212764 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.212765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.212766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.212829 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.213111 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.213112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.213113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.213257 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.213419 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.213465 (+ 46us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.213544 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:02:24.213810 30537 raft_consensus.cc:348] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.213932 30537 raft_consensus.cc:729] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 895a3b7f860447f695463c8cc7e14ad2, State: Initialized, Role: FOLLOWER
I20260925 15:02:24.214005 30537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.214120 30687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: Bootstrap starting.
I20260925 15:02:24.214358 30537 sys_catalog.cc:564] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:24.214737 30687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:24.214977 30687 log.cc:824] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:24.215696 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.215804 30687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: No bootstrap required, opened a new log
I20260925 15:02:24.215796 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:02:24.216378 30706 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:24.216434 30706 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:24.217696 30687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.217849 30687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3548877a3a464089bf996171223560, State: Initialized, Role: FOLLOWER
I20260925 15:02:24.217923 30687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.218166 30708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.218327 30708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.218250 30687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:24.217149 30469 master_runner.cc:186] Error getting master registration for 127.8.13.253:39061: OK, instance_id { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" instance_seqno: 1790348544020818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:02:24.219511 30469 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:02:24.220600 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260925 15:02:24.220614 30719 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:24.220655 30719 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:24.221424 30511 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "895a3b7f860447f695463c8cc7e14ad2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55860
I20260925 15:02:24.221539 30511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:02:24.221585 30511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:24.222527 30511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.222766 30511 leader_election.cc:288] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [CANDIDATE]: Term 1 election: Requested vote from peers bc4cdce86741462597ff714c1f1faca8 (127.8.13.253:39061), 7d3548877a3a464089bf996171223560 (127.8.13.252:38455)
I20260925 15:02:24.223100 30661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "895a3b7f860447f695463c8cc7e14ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7d3548877a3a464089bf996171223560"
I20260925 15:02:24.223218 30661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:24.221611 30619 master_runner.cc:186] Error getting master registration for 127.8.13.253:39061: OK, instance_id { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" instance_seqno: 1790348544020818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:02:24.223896 30619 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:02:24.224135 30661 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895a3b7f860447f695463c8cc7e14ad2 in term 1.
I20260925 15:02:24.224330 30486 leader_election.cc:302] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3548877a3a464089bf996171223560, 895a3b7f860447f695463c8cc7e14ad2; no voters:
I20260925 15:02:24.224490 30694 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:24.224725 30694 raft_consensus.cc:686] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [term 1 LEADER]: Becoming Leader. State: Replica: 895a3b7f860447f695463c8cc7e14ad2, State: Running, Role: LEADER
I20260925 15:02:24.224835 30694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.225317 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895a3b7f860447f695463c8cc7e14ad2. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.225423 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:24.225721 30727 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:24.226101 30727 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:24.228355 30661 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [term 1 FOLLOWER]: Refusing update from remote peer 895a3b7f860447f695463c8cc7e14ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:24.229139 30722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:24.230834 30708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895a3b7f860447f695463c8cc7e14ad2. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.230954 30708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.232075 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.232177 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:24.232414 30708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.232504 30708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.232831 30729 mvcc.cc:205] Tried to move back new op lower bound from 7333267637156384768 to 7333267637145714688. Current Snapshot: MvccSnapshot[applied={T|T < 7333267637156384768}]
I20260925 15:02:24.242390 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.232056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35744 (local address 127.8.13.253:39061)
0925 15:02:24.232108 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.232110 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.232785 (+ 675us) server_negotiation.cc:407] Connection header received
0925 15:02:24.233002 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.233004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.233015 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.233042 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.234190 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.234932 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.236029 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.236349 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.236391 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.238510 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.238511 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.238512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.238519 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.240656 (+ 2137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.241041 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.241043 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.241043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.241098 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.241751 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.241752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.241752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.241902 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.241987 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.242207 (+ 220us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.242286 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:02:24.242938 30727 catalog_manager.cc:1347] Generated new cluster ID: b489481539d0480297435493ac07ccca
I20260925 15:02:24.242993 30727 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260925 15:02:24.243158 30540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.225328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55862 (local address 127.8.13.254:42267)
0925 15:02:24.225367 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.225369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.232758 (+ 7389us) server_negotiation.cc:407] Connection header received
0925 15:02:24.232979 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.232980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.232993 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.233038 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.234294 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.235028 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.236801 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.237117 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.237155 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.239425 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.239426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.239426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.239434 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.241037 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.241706 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.241708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.241708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.241780 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.242084 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.242084 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.242085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.242233 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.242309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.242992 (+ 683us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.243078 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:02:24.253640 30612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } attempt: 3
I20260925 15:02:24.254452 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.232239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50404 (local address 127.8.13.252:38455)
0925 15:02:24.232275 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.232277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.242741 (+ 10464us) server_negotiation.cc:407] Connection header received
0925 15:02:24.242770 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.242772 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.242787 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.242835 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.244774 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.245587 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.247291 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.247734 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.247779 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.250175 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.250177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.250177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.250186 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.252027 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.252844 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.252845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.252860 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.252920 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.253220 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.253220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.253221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.253382 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.253456 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.254284 (+ 828us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.254361 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260925 15:02:24.255731 30727 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:24.256548 30727 catalog_manager.cc:1504] Loading token signing keys...
Sep 25 15:02:24 dist-test-slave-15hz krb5kdc[30456](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348543, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:24.259997 30727 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2: Generated new TSK 0
I20260925 15:02:24.260201 30727 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260925 15:02:24.261785 30690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.253984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50410 (local address 127.8.13.252:38455)
0925 15:02:24.254153 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.254155 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.254161 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:24.254179 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.254180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.254189 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.254231 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.254596 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.255246 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.256502 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.256781 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.256846 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.259041 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:24.259043 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:24.259043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:24.259050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:24.260634 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.260972 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.260973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.260974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.261055 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:24.261261 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:24.261262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:24.261262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:24.261408 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:24.261475 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:24.261612 (+ 137us) server_negotiation.cc:299] Negotiation successful
0925 15:02:24.261696 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260925 15:02:24.261787 30614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:24.253958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:38455 (local address 127.0.0.1:50410)
0925 15:02:24.254025 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:24.254035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:24.254078 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:24.254275 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:24.254276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:24.254434 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:24.254479 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.254482 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.255502 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.255503 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.256303 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:24.256306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:24.256860 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:24.256861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:24.256915 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:24.257426 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:24.257432 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:24.258965 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:24.260698 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.260699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.260700 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.260897 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.261125 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:24.261126 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:24.261126 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:24.261204 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:24.261465 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:24.261466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:24.261512 (+ 46us) client_negotiation.cc:769] Sending connection context
0925 15:02:24.261576 (+ 64us) client_negotiation.cc:240] Negotiation successful
0925 15:02:24.261685 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260925 15:02:24.263286 30612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: Bootstrap starting.
I20260925 15:02:24.264009 30612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:24.264279 30612 log.cc:824] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:24.265115 30612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: No bootstrap required, opened a new log
I20260925 15:02:24.267158 30612 raft_consensus.cc:348] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.267298 30612 raft_consensus.cc:729] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc4cdce86741462597ff714c1f1faca8, State: Initialized, Role: FOLLOWER
I20260925 15:02:24.267508 30612 consensus_queue.cc:260] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } }
I20260925 15:02:24.267901 30612 sys_catalog.cc:564] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:24.267995 30585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:24.268062 30586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "895a3b7f860447f695463c8cc7e14ad2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "bc4cdce86741462597ff714c1f1faca8"
I20260925 15:02:24.267897 30737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
W20260925 15:02:24.268754 30565 tablet.cc:2369] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260925 15:02:24.270970 30737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.271499 30585 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [term 1 FOLLOWER]: Refusing update from remote peer 895a3b7f860447f695463c8cc7e14ad2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:24.271978 30694 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:24.273465 30586 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 895a3b7f860447f695463c8cc7e14ad2 in term 1.
W20260925 15:02:24.274416 30749 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:24.274516 30749 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:24.275112 30737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 895a3b7f860447f695463c8cc7e14ad2. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.275183 30737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.275211 30737 sys_catalog.cc:455] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.275249 30737 sys_catalog.cc:458] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:24.275895 30751 mvcc.cc:205] Tried to move back new op lower bound from 7333267637283958784 to 7333267637145714688. Current Snapshot: MvccSnapshot[applied={T|T < 7333267637156384768}]
W20260925 15:02:24.292984 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.291327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35756 (local address 127.8.13.253:39061)
0925 15:02:24.291374 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.291377 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.292247 (+ 870us) server_negotiation.cc:407] Connection header received
0925 15:02:24.292371 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.292372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.292385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.292414 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.292905 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35756: BlockingRecv error: recv got EOF from 127.0.0.1:35756 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:02:24.293928 30690 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.291774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50420 (local address 127.8.13.252:38455)
0925 15:02:24.291818 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.291819 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.293398 (+ 1579us) server_negotiation.cc:407] Connection header received
0925 15:02:24.293464 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.293465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.293479 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.293528 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.293847 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50420: BlockingRecv error: recv got EOF from 127.0.0.1:50420 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:02:24.294143 30540 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.291979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55868 (local address 127.8.13.254:42267)
0925 15:02:24.292018 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.292020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.293576 (+ 1556us) server_negotiation.cc:407] Connection header received
0925 15:02:24.293709 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.293711 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.293722 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.293751 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.294082 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55868: BlockingRecv error: recv got EOF from 127.0.0.1:55868 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:02:24.296461 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.295739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35764 (local address 127.8.13.253:39061)
0925 15:02:24.295774 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.295775 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.295957 (+ 182us) server_negotiation.cc:407] Connection header received
0925 15:02:24.296061 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.296062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.296082 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.296110 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.296411 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35764: BlockingRecv error: recv got EOF from 127.0.0.1:35764 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:02:24.299516 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.298808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35780 (local address 127.8.13.253:39061)
0925 15:02:24.298841 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.298842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.299055 (+ 213us) server_negotiation.cc:407] Connection header received
0925 15:02:24.299144 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.299145 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.299156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.299183 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.299470 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35780: BlockingRecv error: recv got EOF from 127.0.0.1:35780 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260925 15:02:24.303507 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.302784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35786 (local address 127.8.13.253:39061)
0925 15:02:24.302819 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.302821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.303028 (+ 207us) server_negotiation.cc:407] Connection header received
0925 15:02:24.303118 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.303119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.303129 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.303157 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.303459 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35786: BlockingRecv error: recv got EOF from 127.0.0.1:35786 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260925 15:02:24.309787 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.308954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35790 (local address 127.8.13.253:39061)
0925 15:02:24.308993 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.308996 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.309247 (+ 251us) server_negotiation.cc:407] Connection header received
0925 15:02:24.309345 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.309346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.309358 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.309384 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.309730 (+ 346us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35790: BlockingRecv error: recv got EOF from 127.0.0.1:35790 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260925 15:02:24.320236 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.319257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35804 (local address 127.8.13.253:39061)
0925 15:02:24.319302 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.319304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.319604 (+ 300us) server_negotiation.cc:407] Connection header received
0925 15:02:24.319706 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.319708 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.319720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.319748 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.320167 (+ 419us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35804: BlockingRecv error: recv got EOF from 127.0.0.1:35804 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:02:24.338692 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.337741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35810 (local address 127.8.13.253:39061)
0925 15:02:24.337786 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.337788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.338070 (+ 282us) server_negotiation.cc:407] Connection header received
0925 15:02:24.338179 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.338180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.338192 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.338221 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.338630 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35810: BlockingRecv error: recv got EOF from 127.0.0.1:35810 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:02:24.373302 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.372337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35816 (local address 127.8.13.253:39061)
0925 15:02:24.372384 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.372386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.372656 (+ 270us) server_negotiation.cc:407] Connection header received
0925 15:02:24.372750 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.372752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.372764 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.372790 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.373240 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35816: BlockingRecv error: recv got EOF from 127.0.0.1:35816 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260925 15:02:24.439605 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.438915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35822 (local address 127.8.13.253:39061)
0925 15:02:24.438966 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.438967 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.439107 (+ 140us) server_negotiation.cc:407] Connection header received
0925 15:02:24.439144 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.439146 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.439159 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.439216 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.439519 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35822: BlockingRecv error: recv got EOF from 127.0.0.1:35822 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260925 15:02:24.569203 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.568627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35834 (local address 127.8.13.253:39061)
0925 15:02:24.568690 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.568691 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.568761 (+ 70us) server_negotiation.cc:407] Connection header received
0925 15:02:24.568793 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.568794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.568805 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.568862 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.569149 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35834: BlockingRecv error: recv got EOF from 127.0.0.1:35834 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:02:24.795418 30810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "895a3b7f860447f695463c8cc7e14ad2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "895a3b7f860447f695463c8cc7e14ad2" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 42267 } } peers { permanent_uuid: "bc4cdce86741462597ff714c1f1faca8" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39061 } } peers { permanent_uuid: "7d3548877a3a464089bf996171223560" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 38455 } } }
I20260925 15:02:24.795600 30810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 895a3b7f860447f695463c8cc7e14ad2 [sys.catalog]: This master's current role is: LEADER
W20260925 15:02:24.826897 30616 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:24.826370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35840 (local address 127.8.13.253:39061)
0925 15:02:24.826430 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:24.826432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:24.826443 (+ 11us) server_negotiation.cc:407] Connection header received
0925 15:02:24.826469 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:24.826470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:24.826480 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:24.826528 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:24.826833 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35840: BlockingRecv error: recv got EOF from 127.0.0.1:35840 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260925 15:02:25.221088 30719 catalog_manager.cc:1259] Loaded cluster ID: b489481539d0480297435493ac07ccca
I20260925 15:02:25.221169 30719 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: loading cluster ID for follower catalog manager: success
I20260925 15:02:25.221932 30719 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: acquiring CA information for follower catalog manager: success
I20260925 15:02:25.222190 30719 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7d3548877a3a464089bf996171223560: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:02:25.274956 30749 catalog_manager.cc:1259] Loaded cluster ID: b489481539d0480297435493ac07ccca
I20260925 15:02:25.275035 30749 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: loading cluster ID for follower catalog manager: success
I20260925 15:02:25.275971 30749 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: acquiring CA information for follower catalog manager: success
I20260925 15:02:25.276294 30749 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bc4cdce86741462597ff714c1f1faca8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:02:25.342574 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:25.340100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35842 (local address 127.8.13.253:39061)
0925 15:02:25.340212 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:25.340214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:25.340239 (+ 25us) server_negotiation.cc:407] Connection header received
0925 15:02:25.340318 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:25.340319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:25.340384 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:25.340448 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:25.340581 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.341074 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.341836 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.342040 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.342099 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:25.342246 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:25.342358 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:25.342425 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:25.342463 (+ 38us) server_negotiation.cc:299] Negotiation successful
0925 15:02:25.342485 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":59,"threads_started":1}
I20260925 15:02:25.346514 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:25.344267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50430 (local address 127.8.13.252:38455)
0925 15:02:25.344361 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:25.344363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:25.344370 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:02:25.344404 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:25.344405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:25.344438 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:25.344505 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:25.344644 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.345110 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.345843 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.346035 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.346081 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:25.346240 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:25.346341 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:25.346391 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:25.346420 (+ 29us) server_negotiation.cc:299] Negotiation successful
0925 15:02:25.346438 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":50,"threads_started":1}
I20260925 15:02:25.350636 30823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:25.348053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35858 (local address 127.8.13.253:39061)
0925 15:02:25.348092 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:25.348094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:25.348229 (+ 135us) server_negotiation.cc:407] Connection header received
0925 15:02:25.348336 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:25.348337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:25.348350 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:25.348402 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:25.348511 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.349245 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350011 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.350244 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350301 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:25.350413 (+ 112us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:25.350444 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:25.350486 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:25.350526 (+ 40us) server_negotiation.cc:299] Negotiation successful
0925 15:02:25.350548 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260925 15:02:25.351012 30829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:25.348310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50446 (local address 127.8.13.252:38455)
0925 15:02:25.348353 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:25.348354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:25.348398 (+ 44us) server_negotiation.cc:407] Connection header received
0925 15:02:25.348470 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:25.348471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:25.348480 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:25.348526 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:25.348687 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.349147 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350190 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.350437 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350492 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:25.350708 (+ 216us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:25.350742 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:25.350812 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:25.350922 (+ 110us) server_negotiation.cc:299] Negotiation successful
0925 15:02:25.350947 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260925 15:02:25.351171 30836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:25.348021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55884 (local address 127.8.13.254:42267)
0925 15:02:25.348122 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:25.348125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:25.348255 (+ 130us) server_negotiation.cc:407] Connection header received
0925 15:02:25.348304 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:25.348305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:25.348358 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:25.348425 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:25.348588 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.349328 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350268 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:25.350568 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:25.350638 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:25.350812 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:25.350928 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:25.350979 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:25.351058 (+ 79us) server_negotiation.cc:299] Negotiation successful
0925 15:02:25.351083 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":51,"threads_started":1}
I20260925 15:02:25.360435 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30469
I20260925 15:02:25.365500 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30544
I20260925 15:02:25.369751 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30619
2026-09-25T15:02:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2695 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 25 15:02:25 dist-test-slave-15hz krb5kdc[30840](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 25 15:02:25 dist-test-slave-15hz krb5kdc[30840](info): set up 2 sockets
Sep 25 15:02:25 dist-test-slave-15hz krb5kdc[30840](info): commencing operation
krb5kdc: starting...
W20260925 15:02:26.323451 8247 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/test-user.keytab.
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-25T15:02:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-25T15:02:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.13.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
Entry for principal kudu/127.8.13.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab.
I20260925 15:02:26.435050 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37985
--webserver_interface=127.8.13.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260925 15:02:26.514374 30853 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:26.514613 30853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:26.514631 30853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:26.514652 30853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:26.515825 30853 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:26.515865 30853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:26.515877 30853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:26.515888 30853 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:26.515899 30853 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:26.517391 30853 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.254.keytab
--principal=kudu/127.8.13.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.254:37985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.13.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:26.517608 30853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:26.517795 30853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:26.522420 30861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:26.522445 30853 server_base.cc:1034] running on GCE node
W20260925 15:02:26.522420 30858 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:26.522451 30859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:26.522855 30853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:26.523129 30853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:26.524271 30853 hybrid_clock.cc:648] HybridClock initialized: now 1790348546524258 us; error 28 us; skew 500 ppm
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:26.526612 30853 init.cc:377] Logged in from keytab as kudu/127.8.13.254@KRBTEST.COM (short username kudu)
I20260925 15:02:26.527040 30853 webserver.cc:460] Webserver started at http://127.8.13.254:46407/ using document root <none> and password file <none>
I20260925 15:02:26.527242 30853 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:26.527280 30853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:26.527369 30853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:26.528393 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data/instance:
uuid: "1968003d9cfd4d78a5f284032b6be5a8"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "df85d3d2132443a3db8a30d58c55c894"
server_key_iv: "ca0aa1384bc742eac33806899f80a058"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.528693 30853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance:
uuid: "1968003d9cfd4d78a5f284032b6be5a8"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "df85d3d2132443a3db8a30d58c55c894"
server_key_iv: "ca0aa1384bc742eac33806899f80a058"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.529904 30853 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260925 15:02:26.530511 30868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:26.530671 30853 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260925 15:02:26.530753 30853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal
uuid: "1968003d9cfd4d78a5f284032b6be5a8"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "df85d3d2132443a3db8a30d58c55c894"
server_key_iv: "ca0aa1384bc742eac33806899f80a058"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.530808 30853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:26.536877 30853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:26.538440 30853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:26.538539 30853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:26.542090 30853 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.254:37985
I20260925 15:02:26.542119 30920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.254:37985 every 8 connection(s)
I20260925 15:02:26.542433 30853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/data/info.pb
I20260925 15:02:26.543360 30921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:26.544915 30921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.549495 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30853
I20260925 15:02:26.549598 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-0/wal/instance
I20260925 15:02:26.549782 8247 external_mini_cluster.cc:1428] Setting key f5aff9f8390e6989f1a01affa67fe2be
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:26.554191 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.545604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45538 (local address 127.8.13.254:37985)
0925 15:02:26.545803 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.545807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.545890 (+ 83us) server_negotiation.cc:407] Connection header received
0925 15:02:26.546406 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.546409 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.546428 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.546511 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.547012 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.547879 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.548712 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.549013 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.549141 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.551167 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.551175 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.551175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.551183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.552653 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.553104 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.553106 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.553107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.553182 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.553406 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.553407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.553407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.553607 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.553662 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.553878 (+ 216us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.554012 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":179,"thread_start_us":75,"threads_started":1}
I20260925 15:02:26.554199 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.545583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37985 (local address 127.0.0.1:45538)
0925 15:02:26.545750 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.545794 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.546264 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.546510 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.546512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.546776 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.546924 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.546929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.547959 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.547960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.548636 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.548638 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.549103 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.549104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.549167 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.549754 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.549762 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.551090 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.552747 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.552755 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.552758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.553026 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.553240 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.553241 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.553242 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.553350 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.553667 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.553673 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.553753 (+ 80us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.553856 (+ 103us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.554082 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":65,"thread_start_us":77,"threads_started":1}
I20260925 15:02:26.555054 30921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.555406 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.555279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:45582)
0925 15:02:26.555327 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.555380 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:02:26.555550 30871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
I20260925 15:02:26.555941 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.555845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:45590)
0925 15:02:26.555889 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.555917 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:02:26.556049 30921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39305: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.13.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
Entry for principal kudu/127.8.13.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab.
I20260925 15:02:26.572495 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:39305
--webserver_interface=127.8.13.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:26.605275 30921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } attempt: 1
I20260925 15:02:26.605809 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.605638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:45602)
0925 15:02:26.605712 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.605765 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260925 15:02:26.606145 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.606057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:45608)
0925 15:02:26.606095 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.606123 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260925 15:02:26.606236 30921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39305: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
W20260925 15:02:26.651480 30928 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:26.651731 30928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:26.651747 30928 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:26.651767 30928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:26.652951 30928 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:26.652997 30928 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:26.653010 30928 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:26.653021 30928 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:26.653033 30928 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:26.654537 30928 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.253.keytab
--principal=kudu/127.8.13.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.253:39305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.13.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30928
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:26.654775 30928 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:26.655002 30928 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:26.659734 30936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:26.659727 30934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:26.659734 30933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:26.659759 30928 server_base.cc:1034] running on GCE node
I20260925 15:02:26.660141 30928 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:26.660458 30928 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:26.661612 30928 hybrid_clock.cc:648] HybridClock initialized: now 1790348546661596 us; error 31 us; skew 500 ppm
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:26.664378 30928 init.cc:377] Logged in from keytab as kudu/127.8.13.253@KRBTEST.COM (short username kudu)
I20260925 15:02:26.664884 30928 webserver.cc:460] Webserver started at http://127.8.13.253:36471/ using document root <none> and password file <none>
I20260925 15:02:26.665103 30928 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:26.665153 30928 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:26.665280 30928 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:26.666332 30928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data/instance:
uuid: "4863480369004238902fc00343d067af"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "6b4db8b7125651869b43d0ed503d19eb"
server_key_iv: "3fd193fb42d9094be01ef219dee5865f"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.666810 30928 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal/instance:
uuid: "4863480369004238902fc00343d067af"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "6b4db8b7125651869b43d0ed503d19eb"
server_key_iv: "3fd193fb42d9094be01ef219dee5865f"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.668560 30928 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260925 15:02:26.669497 30943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:26.669657 30928 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260925 15:02:26.669754 30928 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal
uuid: "4863480369004238902fc00343d067af"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "6b4db8b7125651869b43d0ed503d19eb"
server_key_iv: "3fd193fb42d9094be01ef219dee5865f"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.669831 30928 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:26.674394 30921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } attempt: 2
I20260925 15:02:26.674795 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.674671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:38428)
0925 15:02:26.674723 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.674762 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260925 15:02:26.675051 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.674996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:38434)
0925 15:02:26.675017 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.675037 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260925 15:02:26.675151 30921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.253:39305: Network error: Client connection negotiation failed: client connection to 127.8.13.253:39305: connect: Connection refused (error 111)
I20260925 15:02:26.682171 30928 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:26.684315 30928 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:26.684422 30928 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:26.688105 30928 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.253:39305
I20260925 15:02:26.688148 30995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.253:39305 every 8 connection(s)
I20260925 15:02:26.688455 30928 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/data/info.pb
I20260925 15:02:26.689365 30996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:26.690876 30996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:26.697005 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 30928
I20260925 15:02:26.697085 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-1/wal/instance
I20260925 15:02:26.697274 8247 external_mini_cluster.cc:1428] Setting key 4167929d387c7bacb169fac77a1733c1
I20260925 15:02:26.699554 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.691604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.8.13.254:37985)
0925 15:02:26.691664 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.691666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.691818 (+ 152us) server_negotiation.cc:407] Connection header received
0925 15:02:26.692307 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.692308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.692323 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.692375 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.692977 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.693458 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.694407 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.694609 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.694665 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.696691 (+ 2026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.696692 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.696693 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.696701 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.698251 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.698645 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.698647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.698647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.698702 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.698912 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.698913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.698913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.699027 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.699083 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.699348 (+ 265us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.699440 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:02:26.699724 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.691498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37985 (local address 127.0.0.1:42474)
0925 15:02:26.691694 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.691742 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.692214 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.692442 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.692447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.692697 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.692890 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.692894 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.693541 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.693542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.694327 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.694329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.694697 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.694698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.694810 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.695276 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.695284 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.696602 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.698328 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.698335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.698337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.698561 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.698757 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.698758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.698758 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.698856 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.699092 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.699099 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.699229 (+ 130us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.699317 (+ 88us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.699474 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260925 15:02:26.700208 30996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.13.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.13.252@KRBTEST.COM" created.
I20260925 15:02:26.707126 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.700434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:38438)
0925 15:02:26.700472 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.700482 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.700519 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.700764 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.700765 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.700895 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.700937 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.700939 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.701739 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.701740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.702239 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.702240 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.702660 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.702660 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.702724 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.703076 (+ 352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.703081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.704402 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.706112 (+ 1710us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.706113 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.706114 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.706301 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.706524 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.706524 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.706525 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.706583 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.706885 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.706886 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.706923 (+ 37us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.706970 (+ 47us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.707059 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260925 15:02:26.707237 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.700502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38438 (local address 127.8.13.253:39305)
0925 15:02:26.700652 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.700656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.700670 (+ 14us) server_negotiation.cc:407] Connection header received
0925 15:02:26.700685 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.700687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.700697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.700753 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.700996 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.701665 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.702304 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.702600 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.702656 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.704480 (+ 1824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.704488 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.704488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.704494 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.706021 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.706390 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.706393 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.706393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.706462 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.706643 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.706644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.706645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.706827 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.706905 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.707007 (+ 102us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.707120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":125,"thread_start_us":99,"threads_started":1}
I20260925 15:02:26.708184 30996 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.708498 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.708395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54488)
0925 15:02:26.708440 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.708482 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:02:26.708606 30944 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
I20260925 15:02:26.708968 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.708878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54490)
0925 15:02:26.708924 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.708948 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260925 15:02:26.709092 30996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:40023: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
Entry for principal kudu/127.8.13.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab.
I20260925 15:02:26.720404 8247 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskxyjGoh/build/release/bin/kudu
/tmp/dist-test-taskxyjGoh/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:40023
--webserver_interface=127.8.13.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.13.212:40471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260925 15:02:26.758303 30996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } attempt: 1
I20260925 15:02:26.758793 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.758674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54504)
0925 15:02:26.758727 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.758761 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:02:26.759099 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.759020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54510)
0925 15:02:26.759056 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.759079 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260925 15:02:26.759186 30996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:40023: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
I20260925 15:02:26.766289 30921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } attempt: 3
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:02:26.773169 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.766725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38448 (local address 127.8.13.253:39305)
0925 15:02:26.766784 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.766786 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.766794 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:02:26.766822 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.766823 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.766835 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.766881 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.767210 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.767765 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.768465 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.768693 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.768750 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.770769 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.770771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.770771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.770779 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.772103 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.772454 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.772456 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.772456 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.772511 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.772677 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.772678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.772678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.772791 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.772845 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.772991 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.773070 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260925 15:02:26.773169 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.766615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:38448)
0925 15:02:26.766671 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.766686 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.766731 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.766893 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.766893 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.767064 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.767134 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.767136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.767848 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.767849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.768387 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.768389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.768755 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.768757 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.768833 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.769250 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.769255 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.770679 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.772181 (+ 1502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.772182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.772182 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.772385 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.772566 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.772567 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.772567 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.772627 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.772844 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.772845 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.772895 (+ 50us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.772962 (+ 67us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.773069 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260925 15:02:26.773566 30921 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.773896 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.773788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54524)
0925 15:02:26.773846 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.773873 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:02:26.774161 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.774082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54526)
0925 15:02:26.774116 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.774138 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260925 15:02:26.774220 30921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:40023: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
W20260925 15:02:26.803005 31003 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260925 15:02:26.803236 31003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260925 15:02:26.803252 31003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260925 15:02:26.803272 31003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260925 15:02:26.804533 31003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260925 15:02:26.804572 31003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260925 15:02:26.804585 31003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260925 15:02:26.804605 31003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260925 15:02:26.804629 31003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260925 15:02:26.805341 30921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } attempt: 1
I20260925 15:02:26.805727 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.805608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54530)
0925 15:02:26.805666 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.805696 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:02:26.806093 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.806009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54540)
0925 15:02:26.806049 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.806073 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260925 15:02:26.806178 30921 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:40023: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
I20260925 15:02:26.806290 31003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.13.212:40471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.13.254:37985,127.8.13.253:39305,127.8.13.252:40023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/krb5kdc/kudu_127.8.13.252.keytab
--principal=kudu/127.8.13.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.13.252:40023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.13.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 25 Sep 2026 13:57:57 UTC on ccycloud.quasar-zfbtwj.root.comops.site
build id 4672727
I20260925 15:02:26.806485 31003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260925 15:02:26.806663 31003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260925 15:02:26.811195 31011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:26.811205 31008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260925 15:02:26.811205 31009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260925 15:02:26.811427 31003 server_base.cc:1034] running on GCE node
I20260925 15:02:26.811691 31003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260925 15:02:26.811961 31003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260925 15:02:26.813099 31003 hybrid_clock.cc:648] HybridClock initialized: now 1790348546813084 us; error 35 us; skew 500 ppm
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260925 15:02:26.815454 31003 init.cc:377] Logged in from keytab as kudu/127.8.13.252@KRBTEST.COM (short username kudu)
I20260925 15:02:26.815907 31003 webserver.cc:460] Webserver started at http://127.8.13.252:41981/ using document root <none> and password file <none>
I20260925 15:02:26.816119 31003 fs_manager.cc:359] Metadata directory not provided
I20260925 15:02:26.816166 31003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260925 15:02:26.816287 31003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260925 15:02:26.817322 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data/instance:
uuid: "9d174f06fc4f4bfc815005fab06874bd"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "b7307cd110c3ece8df4f51750636f766"
server_key_iv: "eb5789893c5947e35c27972f985d64c1"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.817658 31003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal/instance:
uuid: "9d174f06fc4f4bfc815005fab06874bd"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "b7307cd110c3ece8df4f51750636f766"
server_key_iv: "eb5789893c5947e35c27972f985d64c1"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.818864 31003 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260925 15:02:26.819584 31018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260925 15:02:26.819733 31003 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260925 15:02:26.819813 31003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data,/tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal
uuid: "9d174f06fc4f4bfc815005fab06874bd"
format_stamp: "Formatted at 2026-09-25 15:02:26 on dist-test-slave-15hz"
server_key: "b7307cd110c3ece8df4f51750636f766"
server_key_iv: "eb5789893c5947e35c27972f985d64c1"
server_key_version: "encryptionkey@0"
I20260925 15:02:26.819877 31003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260925 15:02:26.827329 30996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } attempt: 2
I20260925 15:02:26.827839 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.827706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54542)
0925 15:02:26.827766 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.827807 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260925 15:02:26.828243 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.828164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54556)
0925 15:02:26.828202 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.828226 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260925 15:02:26.828368 30996 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.13.252:40023: Network error: Client connection negotiation failed: client connection to 127.8.13.252:40023: connect: Connection refused (error 111)
I20260925 15:02:26.836074 31003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260925 15:02:26.838157 31003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260925 15:02:26.838276 31003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260925 15:02:26.841949 31003 rpc_server.cc:307] RPC server started. Bound to: 127.8.13.252:40023
I20260925 15:02:26.842006 31070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.13.252:40023 every 8 connection(s)
I20260925 15:02:26.842315 31003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/data/info.pb
I20260925 15:02:26.843257 31071 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260925 15:02:26.844965 8247 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskxyjGoh/build/release/bin/kudu as pid 31003
I20260925 15:02:26.844825 31071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.845064 8247 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskxyjGoh/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790348284423390-8247-0/minicluster-data/master-2/wal/instance
I20260925 15:02:26.845242 8247 external_mini_cluster.cc:1428] Setting key 9d1a56fb3ae9c6c2f5657b5f2c1cdd4c
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:26.853808 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.845588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42478 (local address 127.8.13.254:37985)
0925 15:02:26.845641 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.845643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.845881 (+ 238us) server_negotiation.cc:407] Connection header received
0925 15:02:26.846349 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.846350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.846360 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.846413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.846914 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.847389 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.848416 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.848722 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.848794 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.851072 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.851073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.851074 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.851081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.852511 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.852925 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.852926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.852926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.852985 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.853196 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.853197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.853198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.853315 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.853370 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.853630 (+ 260us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.853715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260925 15:02:26.854039 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.845633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54572 (local address 127.8.13.252:40023)
0925 15:02:26.845783 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.845789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.845823 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:02:26.846274 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.846285 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.846312 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.846396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.846975 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.847976 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.848801 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.849206 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.849263 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.851475 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.851483 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.851483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.851489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.852918 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.853221 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.853224 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.853225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.853281 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.853440 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.853441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.853441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.853584 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.853724 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.853789 (+ 65us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.853864 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260925 15:02:26.854197 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.845575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.254:37985 (local address 127.0.0.1:42478)
0925 15:02:26.845736 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.845781 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.846262 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.846442 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.846451 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.846692 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.846838 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.846843 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.847465 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.847466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.848349 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.848351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.848813 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.848814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.848939 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.849442 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.849450 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.850992 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.852579 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.852587 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.852589 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.852863 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.853035 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.853035 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.853036 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.853142 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.853368 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.853374 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.853511 (+ 137us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.853597 (+ 86us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.853746 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260925 15:02:26.854620 31071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } has no permanent_uuid. Determining permanent_uuid...
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.254@KRBTEST.COM
I20260925 15:02:26.860635 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.854857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.253:39305 (local address 127.0.0.1:38464)
0925 15:02:26.854897 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.854906 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.854937 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.855046 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.855046 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.855179 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.855219 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.855220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.855872 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.855873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.856362 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.856363 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.856687 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.856688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.856753 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.857135 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.857140 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.858345 (+ 1205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.859746 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.859747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.859747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.859903 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.860067 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.860068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.860068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.860122 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.860356 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.860357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.860400 (+ 43us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.860452 (+ 52us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.860529 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260925 15:02:26.860646 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.854919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38464 (local address 127.8.13.253:39305)
0925 15:02:26.854958 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.854960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.854967 (+ 7us) server_negotiation.cc:407] Connection header received
0925 15:02:26.854990 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.854991 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.855000 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.855044 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.855285 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.855811 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.856429 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.856632 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.856683 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.858423 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.858425 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.858426 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.858433 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.859672 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.859961 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.859962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.859962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.860018 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.860169 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.860170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.860170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.860283 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.860345 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.860491 (+ 146us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.860568 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260925 15:02:26.861191 31071 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } has no permanent_uuid. Determining permanent_uuid...
I20260925 15:02:26.861311 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.854904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42494 (local address 127.8.13.254:37985)
0925 15:02:26.854937 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.854939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.854945 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:26.854962 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.854963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.854971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.855015 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.855431 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.855940 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.856670 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.856874 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.856935 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.858975 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.858977 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.858977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.858983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.860244 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.860555 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.860556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.860557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.860611 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.860820 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.860821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.860822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.860950 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.861118 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.861153 (+ 35us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.861223 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.252@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.13.253@KRBTEST.COM
I20260925 15:02:26.867358 31074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.861437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54582)
0925 15:02:26.861480 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.861491 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.861533 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.861671 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.861671 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.861794 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.861828 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.861830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.862527 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.862528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.863030 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.863031 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.863370 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.863370 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.863418 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.863811 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.863815 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.865017 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.866498 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.866499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.866499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.866689 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.866862 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.866862 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.866862 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.866912 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.867153 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.867154 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.867188 (+ 34us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.867234 (+ 46us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.867297 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260925 15:02:26.867453 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.861464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.8.13.252:40023)
0925 15:02:26.861491 (+ 27us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.861493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.861519 (+ 26us) server_negotiation.cc:407] Connection header received
0925 15:02:26.861597 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.861598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.861608 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.861641 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.861895 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.862457 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.863105 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.863310 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.863364 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.865123 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.865124 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.865125 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.865132 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.866438 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.866750 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.866752 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.866752 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.866805 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.866963 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.866964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.866964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.867088 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.867147 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.867271 (+ 124us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.867366 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":14}
I20260925 15:02:26.868736 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.862029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38468 (local address 127.8.13.253:39305)
0925 15:02:26.862061 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.862062 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.862070 (+ 8us) server_negotiation.cc:407] Connection header received
0925 15:02:26.862098 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.862098 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.862107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.862163 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.862600 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.863016 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.863745 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.864050 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.864119 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.866272 (+ 2153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.866273 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.866273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.866279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.867766 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.868057 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.868058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.868058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.868114 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.868255 (+ 141us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.868256 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.868256 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.868370 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.868532 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.868569 (+ 37us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.868651 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260925 15:02:26.868964 31071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: Bootstrap starting.
I20260925 15:02:26.869920 31071 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:26.870146 8247 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260925 15:02:26.870215 31071 log.cc:824] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:26.870919 31071 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: No bootstrap required, opened a new log
I20260925 15:02:26.871146 30895 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1968003d9cfd4d78a5f284032b6be5a8"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42494
I20260925 15:02:26.872437 31071 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.872573 31071 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d174f06fc4f4bfc815005fab06874bd, State: Initialized, Role: FOLLOWER
I20260925 15:02:26.872654 31071 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.872944 31071 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:26.872918 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.873070 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:02:26.874611 31088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:26.874667 31088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:26.874982 31003 master_runner.cc:186] Error getting master registration for 127.8.13.254:37985: OK, instance_id { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" instance_seqno: 1790348546541098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:02:26.877741 31003 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:02:26.881321 30921 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } attempt: 2
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.254@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:26.888054 30923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.881615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54588)
0925 15:02:26.881673 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.881685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.881725 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.881844 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.881845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.882004 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.882048 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.882050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.882658 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.882659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.883167 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.883169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.883493 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.883494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.883579 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.884002 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.884007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.885424 (+ 1417us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.886896 (+ 1472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.886897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.886898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.887125 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.887289 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.887289 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.887289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.887364 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.887713 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.887714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.887769 (+ 55us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.887841 (+ 72us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.887943 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260925 15:02:26.888068 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.881665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54588 (local address 127.8.13.252:40023)
0925 15:02:26.881707 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.881708 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.881718 (+ 10us) server_negotiation.cc:407] Connection header received
0925 15:02:26.881773 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.881774 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.881784 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.881833 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.882114 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.882585 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.883240 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.883434 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.883490 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.885509 (+ 2019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.885511 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.885511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.885520 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.886828 (+ 1308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.887191 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.887192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.887192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.887240 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.887420 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.887421 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.887421 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.887647 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.887716 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.887882 (+ 166us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.887964 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260925 15:02:26.889709 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: Bootstrap starting.
I20260925 15:02:26.890445 30921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:26.890760 30921 log.cc:824] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:26.891651 30921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: No bootstrap required, opened a new log
I20260925 15:02:26.893030 30921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.893184 30921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1968003d9cfd4d78a5f284032b6be5a8, State: Initialized, Role: FOLLOWER
I20260925 15:02:26.893268 30921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.893606 30921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:26.893548 31091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.893673 31091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:26.893733 30895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260925 15:02:26.893783 30895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:26.894491 30895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.894714 30895 leader_election.cc:288] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [CANDIDATE]: Term 1 election: Requested vote from peers 4863480369004238902fc00343d067af (127.8.13.253:39305), 9d174f06fc4f4bfc815005fab06874bd (127.8.13.252:40023)
I20260925 15:02:26.896207 31045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1968003d9cfd4d78a5f284032b6be5a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9d174f06fc4f4bfc815005fab06874bd"
I20260925 15:02:26.896353 31045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [term 0 FOLLOWER]: Advancing to term 1
W20260925 15:02:26.896916 31109 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:26.897025 31109 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:26.897253 31045 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1968003d9cfd4d78a5f284032b6be5a8 in term 1.
I20260925 15:02:26.897488 30869 leader_election.cc:302] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1968003d9cfd4d78a5f284032b6be5a8, 9d174f06fc4f4bfc815005fab06874bd; no voters:
I20260925 15:02:26.897709 31091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 1 FOLLOWER]: Leader election won for term 1
I20260925 15:02:26.897840 31091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [term 1 LEADER]: Becoming Leader. State: Replica: 1968003d9cfd4d78a5f284032b6be5a8, State: Running, Role: LEADER
I20260925 15:02:26.897921 31091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.898320 31091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1968003d9cfd4d78a5f284032b6be5a8. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.898389 31091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:26.899312 30853 master_runner.cc:186] Error getting master registration for 127.8.13.253:39305: OK, instance_id { permanent_uuid: "4863480369004238902fc00343d067af" instance_seqno: 1790348546687039 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260925 15:02:26.901675 30853 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260925 15:02:26.907716 31113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260925 15:02:26.907971 31113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260925 15:02:26.908720 30924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.895881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42510 (local address 127.8.13.254:37985)
0925 15:02:26.895925 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.895927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.896041 (+ 114us) server_negotiation.cc:407] Connection header received
0925 15:02:26.899199 (+ 3158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.899201 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.899213 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.899243 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.900052 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.900804 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.901868 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.902172 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.902243 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.904357 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.904359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.904359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.904368 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.906370 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.907166 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.907167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.907168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.907235 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.908145 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.908146 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.908147 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.908277 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.908343 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.908547 (+ 204us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.908640 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260925 15:02:26.911790 31000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.895800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.8.13.253:39305)
0925 15:02:26.895847 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.895849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.901959 (+ 6110us) server_negotiation.cc:407] Connection header received
0925 15:02:26.902054 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.902055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.902068 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.902098 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.902811 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.903485 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.905241 (+ 1756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.905568 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.905639 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.908007 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.908008 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.908008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.908017 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.909635 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.910220 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.910222 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.910222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.910293 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.910569 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.910570 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.910570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.910717 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.911238 (+ 521us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.911590 (+ 352us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.911691 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260925 15:02:26.914803 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.895986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54594 (local address 127.8.13.252:40023)
0925 15:02:26.896040 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.896042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.903186 (+ 7144us) server_negotiation.cc:407] Connection header received
0925 15:02:26.903300 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.903302 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.903313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.903341 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.906066 (+ 2725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.906791 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.910308 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.910611 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.910652 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.912057 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.912059 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.912059 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.912066 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.913639 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.914013 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.914015 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.914015 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.914090 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.914266 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.914267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.914268 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.914414 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.914512 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.914618 (+ 106us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.914705 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260925 15:02:26.916546 31045 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [term 1 FOLLOWER]: Refusing update from remote peer 1968003d9cfd4d78a5f284032b6be5a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:26.916894 31111 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:26.918515 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1968003d9cfd4d78a5f284032b6be5a8. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.918617 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:26.919482 30996 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } attempt: 3
I20260925 15:02:26.919786 31114 mvcc.cc:205] Tried to move back new op lower bound from 7333267648167714816 to 7333267648094642176. Current Snapshot: MvccSnapshot[applied={T|T < 7333267648167714816}]
I20260925 15:02:26.919903 31111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.919976 31111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:26.919965 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.920040 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:26.920442 31115 mvcc.cc:205] Tried to move back new op lower bound from 7333267648167714816 to 7333267648094642176. Current Snapshot: MvccSnapshot[applied={T|T < 7333267648167714816}]
I20260925 15:02:26.921175 31113 catalog_manager.cc:1347] Generated new cluster ID: 2b8a36a61f174fd99d6821df3c2696aa
I20260925 15:02:26.921222 31113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 25 15:02:26 dist-test-slave-15hz krb5kdc[30840](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790348546, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.13.253@KRBTEST.COM for kudu/127.8.13.252@KRBTEST.COM
I20260925 15:02:26.928478 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.919730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.13.252:40023 (local address 127.0.0.1:54606)
0925 15:02:26.919803 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0925 15:02:26.919813 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0925 15:02:26.919851 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0925 15:02:26.920089 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0925 15:02:26.920090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0925 15:02:26.920249 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0925 15:02:26.920297 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.920299 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.921122 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.921123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.921941 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0925 15:02:26.921943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.922569 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.922570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0925 15:02:26.922644 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.923180 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0925 15:02:26.923185 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0925 15:02:26.924763 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0925 15:02:26.927323 (+ 2560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.927325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.927325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.927561 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.927786 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0925 15:02:26.927786 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0925 15:02:26.927787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0925 15:02:26.927854 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0925 15:02:26.928161 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0925 15:02:26.928162 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0925 15:02:26.928221 (+ 59us) client_negotiation.cc:769] Sending connection context
0925 15:02:26.928278 (+ 57us) client_negotiation.cc:240] Negotiation successful
0925 15:02:26.928375 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260925 15:02:26.928999 31075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:26.919916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54606 (local address 127.8.13.252:40023)
0925 15:02:26.919954 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.919955 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.919961 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:26.919980 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.919980 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.919989 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.920034 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.920379 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.921037 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.922111 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:26.922439 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:26.922497 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:26.924864 (+ 2367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0925 15:02:26.924865 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0925 15:02:26.924865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0925 15:02:26.924872 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0925 15:02:26.927219 (+ 2347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.927644 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.927645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.927645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.927714 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0925 15:02:26.927921 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0925 15:02:26.927922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0925 15:02:26.927923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0925 15:02:26.928071 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0925 15:02:26.928124 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:26.928830 (+ 706us) server_negotiation.cc:299] Negotiation successful
0925 15:02:26.928906 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260925 15:02:26.930928 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: Bootstrap starting.
I20260925 15:02:26.931705 30996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: Neither blocks nor log segments found. Creating new log.
I20260925 15:02:26.932008 30996 log.cc:824] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: Log is configured to *not* fsync() on all Append() calls
I20260925 15:02:26.932479 31113 catalog_manager.cc:1370] Generated new certificate authority record
I20260925 15:02:26.933141 30996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: No bootstrap required, opened a new log
I20260925 15:02:26.935137 30996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.935282 30996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4863480369004238902fc00343d067af, State: Initialized, Role: FOLLOWER
I20260925 15:02:26.935362 30996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } }
I20260925 15:02:26.935716 30996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: configured and running, proceeding with master startup.
I20260925 15:02:26.935847 30969 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [term 0 FOLLOWER]: Advancing to term 1
I20260925 15:02:26.936775 30969 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [term 1 FOLLOWER]: Refusing update from remote peer 1968003d9cfd4d78a5f284032b6be5a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260925 15:02:26.937146 31091 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260925 15:02:26.937511 30970 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1968003d9cfd4d78a5f284032b6be5a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "4863480369004238902fc00343d067af"
I20260925 15:02:26.938326 30970 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1968003d9cfd4d78a5f284032b6be5a8 in term 1.
I20260925 15:02:26.939143 31113 catalog_manager.cc:1504] Loading token signing keys...
I20260925 15:02:26.943135 31135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.943316 31135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:26.943404 31135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1968003d9cfd4d78a5f284032b6be5a8. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.943493 31135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: This master's current role is: FOLLOWER
I20260925 15:02:26.943576 31135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.943729 31135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af [sys.catalog]: This master's current role is: FOLLOWER
W20260925 15:02:26.944276 31131 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260925 15:02:26.944331 31131 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260925 15:02:26.945964 31133 mvcc.cc:205] Tried to move back new op lower bound from 7333267648218832896 to 7333267648094642176. Current Snapshot: MvccSnapshot[applied={T|T < 7333267648167714816}]
I20260925 15:02:26.947692 31091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1968003d9cfd4d78a5f284032b6be5a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1968003d9cfd4d78a5f284032b6be5a8" member_type: VOTER last_known_addr { host: "127.8.13.254" port: 37985 } } peers { permanent_uuid: "4863480369004238902fc00343d067af" member_type: VOTER last_known_addr { host: "127.8.13.253" port: 39305 } } peers { permanent_uuid: "9d174f06fc4f4bfc815005fab06874bd" member_type: VOTER last_known_addr { host: "127.8.13.252" port: 40023 } } }
I20260925 15:02:26.947795 31091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8 [sys.catalog]: This master's current role is: LEADER
I20260925 15:02:26.954747 31113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1968003d9cfd4d78a5f284032b6be5a8: Generated new TSK 0
I20260925 15:02:26.955157 31113 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260925 15:02:26.974730 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.973026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38482 (local address 127.8.13.253:39305)
0925 15:02:26.973071 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.973073 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.974035 (+ 962us) server_negotiation.cc:407] Connection header received
0925 15:02:26.974137 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.974139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.974150 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.974185 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.974655 (+ 470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38482: BlockingRecv error: recv got EOF from 127.0.0.1:38482 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260925 15:02:26.975493 30924 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.973428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42514 (local address 127.8.13.254:37985)
0925 15:02:26.973472 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.973474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.974991 (+ 1517us) server_negotiation.cc:407] Connection header received
0925 15:02:26.975086 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.975087 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.975099 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.975125 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.975427 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42514: BlockingRecv error: recv got EOF from 127.0.0.1:42514 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:02:26.976255 31075 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.973724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.8.13.252:40023)
0925 15:02:26.973779 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.973780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.975730 (+ 1950us) server_negotiation.cc:407] Connection header received
0925 15:02:26.975852 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.975854 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.975865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.975893 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.976196 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54616: BlockingRecv error: recv got EOF from 127.0.0.1:54616 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260925 15:02:26.978716 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.977475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38488 (local address 127.8.13.253:39305)
0925 15:02:26.977514 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.977515 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.978158 (+ 643us) server_negotiation.cc:407] Connection header received
0925 15:02:26.978272 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.978274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.978294 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.978322 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.978661 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38488: BlockingRecv error: recv got EOF from 127.0.0.1:38488 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:02:26.981845 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.981017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38504 (local address 127.8.13.253:39305)
0925 15:02:26.981055 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.981057 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.981312 (+ 255us) server_negotiation.cc:407] Connection header received
0925 15:02:26.981409 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.981410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.981422 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.981451 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.981790 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38504: BlockingRecv error: recv got EOF from 127.0.0.1:38504 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:02:26.985880 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.985006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38516 (local address 127.8.13.253:39305)
0925 15:02:26.985046 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.985048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.985306 (+ 258us) server_negotiation.cc:407] Connection header received
0925 15:02:26.985407 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.985408 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.985420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.985451 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.985822 (+ 371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38516: BlockingRecv error: recv got EOF from 127.0.0.1:38516 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260925 15:02:26.992236 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:26.991189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.8.13.253:39305)
0925 15:02:26.991235 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:26.991237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:26.991576 (+ 339us) server_negotiation.cc:407] Connection header received
0925 15:02:26.991694 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:26.991695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:26.991708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:26.991740 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:26.992171 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38518: BlockingRecv error: recv got EOF from 127.0.0.1:38518 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260925 15:02:27.002575 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.001584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38530 (local address 127.8.13.253:39305)
0925 15:02:27.001630 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.001632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.001934 (+ 302us) server_negotiation.cc:407] Connection header received
0925 15:02:27.002044 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.002045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.002059 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.002090 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.002511 (+ 421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38530: BlockingRecv error: recv got EOF from 127.0.0.1:38530 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:02:27.021024 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.019975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.8.13.253:39305)
0925 15:02:27.020036 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.020038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.020352 (+ 314us) server_negotiation.cc:407] Connection header received
0925 15:02:27.020463 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.020464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.020479 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.020511 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.020944 (+ 433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260925 15:02:27.054936 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.054274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38552 (local address 127.8.13.253:39305)
0925 15:02:27.054319 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.054321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.054428 (+ 107us) server_negotiation.cc:407] Connection header received
0925 15:02:27.054492 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.054493 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.054505 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.054555 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.054871 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38552: BlockingRecv error: recv got EOF from 127.0.0.1:38552 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260925 15:02:27.120715 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.120021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38556 (local address 127.8.13.253:39305)
0925 15:02:27.120068 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.120070 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.120143 (+ 73us) server_negotiation.cc:407] Connection header received
0925 15:02:27.120222 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.120223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.120232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.120278 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.120652 (+ 374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38556: BlockingRecv error: recv got EOF from 127.0.0.1:38556 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260925 15:02:27.250618 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.249890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38566 (local address 127.8.13.253:39305)
0925 15:02:27.249987 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.249989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.250092 (+ 103us) server_negotiation.cc:407] Connection header received
0925 15:02:27.250152 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.250153 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.250167 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.250218 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.250548 (+ 330us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38566: BlockingRecv error: recv got EOF from 127.0.0.1:38566 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260925 15:02:27.508598 31000 negotiation.cc:337] Failed RPC negotiation. Trace:
0925 15:02:27.507912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38568 (local address 127.8.13.253:39305)
0925 15:02:27.507991 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:27.507993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:27.508005 (+ 12us) server_negotiation.cc:407] Connection header received
0925 15:02:27.508098 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:27.508099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:27.508112 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:27.508167 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0925 15:02:27.508533 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38568: BlockingRecv error: recv got EOF from 127.0.0.1:38568 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260925 15:02:27.875120 31088 catalog_manager.cc:1259] Loaded cluster ID: 2b8a36a61f174fd99d6821df3c2696aa
I20260925 15:02:27.875197 31088 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: loading cluster ID for follower catalog manager: success
I20260925 15:02:27.876050 31088 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: acquiring CA information for follower catalog manager: success
I20260925 15:02:27.876375 31088 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d174f06fc4f4bfc815005fab06874bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:02:27.944748 31131 catalog_manager.cc:1259] Loaded cluster ID: 2b8a36a61f174fd99d6821df3c2696aa
I20260925 15:02:27.944825 31131 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: loading cluster ID for follower catalog manager: success
I20260925 15:02:27.945637 31131 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: acquiring CA information for follower catalog manager: success
I20260925 15:02:27.945923 31131 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4863480369004238902fc00343d067af: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260925 15:02:28.024348 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:28.021653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.8.13.253:39305)
0925 15:02:28.021783 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:28.021785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:28.021819 (+ 34us) server_negotiation.cc:407] Connection header received
0925 15:02:28.021886 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:28.021887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:28.021941 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:28.022012 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:28.022147 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.022681 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.023470 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.023729 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.023790 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:28.023992 (+ 202us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:28.024113 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:28.024185 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:28.024228 (+ 43us) server_negotiation.cc:299] Negotiation successful
0925 15:02:28.024250 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260925 15:02:28.028052 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:28.025958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.8.13.252:40023)
0925 15:02:28.026063 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:28.026065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:28.026082 (+ 17us) server_negotiation.cc:407] Connection header received
0925 15:02:28.026120 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:28.026120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:28.026155 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:28.026216 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:28.026281 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.026745 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.027382 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.027598 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.027643 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:28.027787 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:28.027888 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:28.027939 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:28.027950 (+ 11us) server_negotiation.cc:299] Negotiation successful
0925 15:02:28.027968 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":62,"threads_started":1}
I20260925 15:02:28.032228 31205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:28.029578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.8.13.253:39305)
0925 15:02:28.029620 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:28.029622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:28.029713 (+ 91us) server_negotiation.cc:407] Connection header received
0925 15:02:28.029771 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:28.029773 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:28.029788 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:28.029838 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:28.030121 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.030581 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.031564 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.031808 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.031862 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:28.031989 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:28.032022 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:28.032076 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:28.032126 (+ 50us) server_negotiation.cc:299] Negotiation successful
0925 15:02:28.032147 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260925 15:02:28.032387 31211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:28.029833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54636 (local address 127.8.13.252:40023)
0925 15:02:28.029884 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:28.029886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:28.029892 (+ 6us) server_negotiation.cc:407] Connection header received
0925 15:02:28.029914 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:28.029915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:28.029922 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:28.029964 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:28.030092 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.030813 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.031668 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.031945 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.031998 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:28.032156 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:28.032186 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:28.032213 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:28.032294 (+ 81us) server_negotiation.cc:299] Negotiation successful
0925 15:02:28.032315 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260925 15:02:28.033036 31218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0925 15:02:28.029548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.8.13.254:37985)
0925 15:02:28.029645 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0925 15:02:28.029647 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0925 15:02:28.029682 (+ 35us) server_negotiation.cc:407] Connection header received
0925 15:02:28.029804 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0925 15:02:28.029806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0925 15:02:28.029863 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0925 15:02:28.029935 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0925 15:02:28.030033 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.030814 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.032010 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0925 15:02:28.032292 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0925 15:02:28.032472 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0925 15:02:28.032704 (+ 232us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0925 15:02:28.032798 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0925 15:02:28.032843 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0925 15:02:28.032925 (+ 82us) server_negotiation.cc:299] Negotiation successful
0925 15:02:28.032953 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78,"thread_start_us":46,"threads_started":1}
I20260925 15:02:28.034466 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30853
I20260925 15:02:28.039392 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 30928
I20260925 15:02:28.043749 8247 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskxyjGoh/build/release/bin/kudu with pid 31003
2026-09-25T15:02:28Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2673 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5369 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (263626 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST